Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-06-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 stickyboy Hmm, interesting notes about replace brick in 3.3.0+
00:00 stickyboy https://lists.gnu.org/archive/html/​gluster-devel/2012-10/msg00050.html
00:00 TheDingy Hummm.... that is new than I have dealt with
00:01 TheDingy and I would go with that
00:03 koguma joined #gluster
00:30 diegows joined #gluster
00:31 stickyboy TheDingy: I think I might have gotten it...
00:31 stickyboy I just mounted the new brick to the old path.
00:32 stickyboy Then set the volume's trusted-gfid filesystem attribute to be correct
00:32 stickyboy Then restarted glusterd
00:32 stickyboy Now it realized it needs to be healed...
00:40 glusterbot New news from newglusterbugs: [Bug 1109613] gluster volume create fails with ambiguous error <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109613>
00:40 koguma joined #gluster
00:41 koguma Filed Bug 1109613
00:41 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1109613 urgent, unspecified, ---, rwheeler, NEW , gluster volume create fails with ambiguous error
00:54 koguma_ joined #gluster
00:54 haomaiwa_ joined #gluster
00:59 bala joined #gluster
01:40 TheDingy stickyboy: glad to hear that you got it
01:46 gildub joined #gluster
02:01 dusmantkp__ joined #gluster
02:23 bala joined #gluster
02:26 gildub joined #gluster
02:29 wgao joined #gluster
02:32 XpineX joined #gluster
02:32 haomaiw__ joined #gluster
03:09 rastar joined #gluster
03:10 kshlm joined #gluster
03:20 diegows joined #gluster
03:23 jag3773 joined #gluster
03:24 coredump joined #gluster
03:34 koguma_ Is there a way to bind the gluster transport to a private ip and localhost?
03:41 rejy joined #gluster
03:42 harish joined #gluster
03:44 MacWinner joined #gluster
03:47 itisravi joined #gluster
03:55 nbalachandran joined #gluster
04:02 kanagaraj joined #gluster
04:06 JoeJulian koguma_: It binds to 0.0.0.0. If you need to block a public ip, use iptables.
04:07 JoeJulian ... or use allow/block rules
04:08 RameshN joined #gluster
04:08 kdhananjay joined #gluster
04:14 kumar joined #gluster
04:16 shubhendu_ joined #gluster
04:23 bharata-rao joined #gluster
04:24 spandit joined #gluster
04:45 haomaiwang joined #gluster
04:46 deepakcs joined #gluster
04:47 koguma_ I figured out a better way.. don't understand why it's not really documented.. had to really google the crap out of it... :)
04:47 koguma_ I set option transport.socket.bind-address <private ip>
04:47 koguma_ Then, you'll get the connection error on the cli
04:48 koguma_ So you just add --remote-host= to gluster cli and bam!
04:48 koguma_ The allow/block rules are good too.  But won't stop someone ddosing your glusterd...
04:51 koguma_ Also, I finally settled my other problems as well.  Apparently 3.4.3 is lax in repoting errors properly..
04:53 ramteid joined #gluster
04:55 rjoseph joined #gluster
04:57 ndarshan joined #gluster
05:00 haomaiw__ joined #gluster
05:03 davinder12 joined #gluster
05:05 ppai joined #gluster
05:11 glusterbot New news from newglusterbugs: [Bug 1075611] [FEAT] log: enhance gluster log format with message ID and standardize errno reporting <https://bugzilla.redhat.co​m/show_bug.cgi?id=1075611>
05:11 koguma_ Hey guys, what is the self heal daemon?
05:13 kdhananjay joined #gluster
05:13 koguma_ 'Self-heal daemon is not running. Check self-heal daemon log file.'
05:16 psharma joined #gluster
05:17 nshaikh joined #gluster
05:18 prasanthp joined #gluster
05:22 koguma_ I see... this all stems from the bind issue...
05:23 koguma_ '/usr/sbin/glusterfs -s localhost .... is the issue...
05:23 lalatenduM joined #gluster
05:23 vimal joined #gluster
05:25 rtalur_ joined #gluster
05:37 koguma_ Is there a command line option to pass to gluster cli for the self-heal daemon?  Apparently --remote-host= doesn't propogate to it...
05:38 hagarth joined #gluster
05:42 karnan joined #gluster
05:44 bala1 joined #gluster
05:51 dusmantkp__ joined #gluster
05:54 rjoseph joined #gluster
05:57 bala1 joined #gluster
06:05 rtalur_ joined #gluster
06:14 raghu joined #gluster
06:19 koguma_ Anyone know how to connect to the self-heal daemon if it's not on localhost?
06:37 calum_ joined #gluster
06:38 koguma_ Can the transport.socket.bind-address be speicified multiple times in the config?
06:38 koguma_ to bind to multiple ip's?
06:40 saurabh joined #gluster
06:41 ramteid koguma_: I'm not sure, I had the problem to, but failed.
06:41 ramteid *too
06:42 ProT-0-TypE joined #gluster
06:42 koguma_ ramteid: seems to not work.. it's such a pain.. it doesn't seem to want to use --remote-host from the cli either...
06:43 ramteid koguma_: can't tell.
06:43 ramteid koguma_: and yes, glusterfs has some rough edges ....
06:47 koguma_ ramteid: Heh yeah.  Trying a few other things.
06:47 ramteid koguma_: I wish u luck on your endavour
06:53 ctria joined #gluster
06:54 haomaiwa_ joined #gluster
06:54 koguma_ ramteid: I'm going to need it.
06:59 hagarth joined #gluster
07:01 rjoseph joined #gluster
07:03 ricky-ti1 joined #gluster
07:05 eseyman joined #gluster
07:06 karnan joined #gluster
07:06 koguma_ ramteid: From this: https://www.mail-archive.com/glust​er-users@gluster.org/msg13529.html it looks like it's possible to have gluster look for the self-heal daemon on another ip....
07:06 glusterbot Title: Re: [Glusterusers] Resync or how to force the replication (at www.mail-archive.com)
07:06 koguma_ just need to find out how...
07:07 ramteid koguma_: unfortunately I have no idea :(
07:08 koguma_ ramteid: seems to be a mystery...
07:08 haomai___ joined #gluster
07:09 vpshastry joined #gluster
07:11 dusmantkp__ joined #gluster
07:16 hybrid512 joined #gluster
07:19 hybrid512 joined #gluster
07:25 rtalur_ joined #gluster
07:34 nishanth joined #gluster
07:34 ktosiek joined #gluster
07:39 deepakcs joined #gluster
07:44 karnan joined #gluster
07:49 Pupeno joined #gluster
07:50 Pupeno How do you guys mount volumes from localhost at boot time?
07:53 Pavid7 joined #gluster
08:03 andreask joined #gluster
08:12 liquidat joined #gluster
08:15 koguma_ Pupeno: I'm about to start doing that.  Going to use automount unless someone has a better way...
08:27 ppai joined #gluster
08:28 ppai Pupeno, through entry in /etc/fstab I guess
08:34 Pupeno ppai: as far as I know, an entry in fstab won't do it.
08:35 Pupeno koguma_: I remember dropping that solution, but not why.
08:35 Pupeno koguma_: what OS are you in?
08:35 koguma_ Pupeno: CentOS 5.x
08:36 koguma_ Pupeno: I'm not sure what other automounting solutions there would be if automount didn't cut it...
08:36 Pupeno I just wish _netdev wrked as expected.
08:37 koguma_ I remember seeing mount examples with _netdev for gluster.  It doesn't work as expected?
08:37 Pupeno koguma_: nope... at least Ubuntu tries to mount them before network is ready. I read _netdev has some nfs hard-coded behaviour, not a generic network behaviour.
08:37 Pupeno That is, _netdev has no effect on non-nfs FSs.
08:40 koguma_ You're using fuse or nfs?
08:40 Pupeno fuse.
08:40 koguma_ Ah, I'm planning on nfs...
08:41 Pupeno Then you can probably just use _netdev and no automount.
08:42 koguma_ Is /usr/sbin/glusterfsd the self heal daemon?
08:42 koguma_ Pupeno: Maybe... yeah, I think you're right... :P
08:43 koguma_ Or is there a specific self-heal daemon?
08:44 Slashman joined #gluster
08:44 Philambdo joined #gluster
08:45 elico joined #gluster
08:49 Thilam joined #gluster
08:53 ramteid koguma_: I grabbed the PID from volume status for SHD, seems to be glusterfs acting as the SHD
08:54 koguma_ ramteid: Thanks, makes sense.  I wonder why it shows Online 'N'...
08:55 ramteid koguma_: that's not good but I pretty sure you know that already
08:55 koguma_ heh yeah
08:55 koguma_ ramteid: glusterfsd is definitely running tho..
08:55 koguma_ I wonder if it relies on the nfs daemon.. which is not running...
08:56 ramteid koguma_: I have no NFS, so I would say no
08:56 koguma_ ramteid: That's no good... :P
09:01 karnan joined #gluster
09:10 mbukatov joined #gluster
09:12 dusmant joined #gluster
09:24 koguma_ When I remove the bind options the self heal daemon is detected on localhost, but not when bind is used.. even on the ip it's bound to...
09:27 rtalur_ joined #gluster
09:37 shubhendu_ joined #gluster
09:38 tryggvil joined #gluster
09:42 Pupeno I posted everything about my mount-at-boot problem here: http://serverfault.com/questions/604860​/glusterfs-is-failing-to-mount-on-boot
09:42 glusterbot Title: ubuntu - GlusterFS is failing to mount on boot - Server Fault (at serverfault.com)
09:57 shubhendu_ joined #gluster
10:06 ppai joined #gluster
10:06 harish joined #gluster
10:09 davidhadas_ joined #gluster
10:12 glusterbot New news from newglusterbugs: [Bug 1109741] glusterd operating version falls back to the operating version of an invalid friend <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109741> || [Bug 1109748] gluster does not detect self-heal daemon when bind is used. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109748>
10:13 haomaiwang joined #gluster
10:16 bharata-rao joined #gluster
10:18 glusterbot New news from resolvedglusterbugs: [Bug 1019874] Glusterfs crash when remove brick while data are still written <https://bugzilla.redhat.co​m/show_bug.cgi?id=1019874> || [Bug 887272] glusterfs process crashed <https://bugzilla.redhat.com/show_bug.cgi?id=887272>
10:23 haomaiwa_ joined #gluster
10:25 haomai___ joined #gluster
10:38 edward1 joined #gluster
10:41 mbukatov joined #gluster
10:42 glusterbot New news from newglusterbugs: [Bug 1109770] [SNAPSHOT] USS: snapview-daemon of a volume should be stopped when the volume is stopped. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109770> || [Bug 1022510] GlusterFS client crashes during add-brick and rebalance <https://bugzilla.redhat.co​m/show_bug.cgi?id=1022510>
10:55 ProT-0-TypE joined #gluster
10:56 rtalur_ joined #gluster
10:57 tdasilva joined #gluster
10:58 nshaikh joined #gluster
10:58 rjoseph joined #gluster
10:59 calum_ joined #gluster
11:02 davidhadas__ joined #gluster
11:14 hchiramm_ joined #gluster
11:16 kkeithley joined #gluster
11:19 kkeithley joined #gluster
11:23 ppai joined #gluster
11:26 davinder13 joined #gluster
11:31 qdk joined #gluster
11:35 dusmant joined #gluster
11:52 diegows joined #gluster
11:53 LebedevRI joined #gluster
12:04 bfoster joined #gluster
12:12 ndk joined #gluster
12:15 tdasilva joined #gluster
12:21 lalatenduM semiosis kkeithley anybody you know maintaining glusterfs packages in debian stable, testing and unstable?
12:21 lalatenduM hagarth, ^^
12:21 morse joined #gluster
12:23 tryggvil joined #gluster
12:24 lalatenduM kkeithley, do you know create debian packages for download.gluster.org is it semiosis ?
12:25 lalatenduM s/know/know who/
12:25 glusterbot What lalatenduM meant to say was: kkeithley, do you know who create debian packages for download.gluster.org is it semiosis ?
12:25 spiekey joined #gluster
12:25 spiekey Hello!
12:25 glusterbot spiekey: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
12:26 spiekey does someone know whats the problem here?
12:26 spiekey http://fpaste.org/110124/02921524/
12:26 glusterbot Title: #110124 Fedora Project Pastebin (at fpaste.org)
12:27 kkeithley lalatenduM: I think semiosis only does Ubuntu .debs. He, or johnmark maybe, knows who does the Debian .debs.
12:28 lalatenduM kkeithley, thanks
12:28 kkeithley IMHO though, we ought to install Debian on some VMs and start doing it ourselves. It's not easy though. (I did it once.)
12:30 lalatenduM kkeithley, yeah debian is difficult, but we should do for debian as it falls in to similar bracket as CentOS wrt to life-cycle and actual production use . May be JoeJulian will help us, he works with debian now :)
12:31 kkeithley does he? poor guy. ;-)
12:31 lalatenduM kkeithley, yeah , he recently changed his job :)
12:31 kkeithley semiosis is the expert on building .debs though
12:32 kkeithley AFAIK
12:32 spiekey hmm…node01 seems to listen on port 49153, and node02 want to connect on port 24007?
12:32 lalatenduM kkeithley, yeah agree
12:33 lalatenduM spiekey, what is actual issue ? looks like some ovirt + gluster thing , I have never used ovirt with gluster
12:33 hchiramm_ joined #gluster
12:33 spiekey lalatenduM: well, currently i am trying to bring up my gluster on node02.
12:34 spiekey with glusterd  -N --debug
12:34 lalatenduM spiekey, yeah I guessed that looking at ur log "0-graph: init failed"
12:36 lalatenduM spiekey, did you change the vol file manually or did some volume set  commands which changed the vol file recently
12:37 kasturi joined #gluster
12:38 spiekey lalatenduM: hmmm…i dont think so. i played around with my mount points, but they should be okay now.
12:38 spiekey whats my problem now?
12:38 hagarth lalatenduM: pmathai used to do debian builds for glusterfs
12:39 sjm joined #gluster
12:39 hagarth lalatenduM: I can share his contact details if you need that
12:39 spiekey Unknown key: brick-0 ?
12:39 lalatenduM spiekey, what is the command you are running now? just "gluster v start"?
12:39 lalatenduM hagarth, thanks
12:40 jag3773 joined #gluster
12:40 lalatenduM spiekey, did you change the partition name or something ?
12:40 lalatenduM I mean brick partition
12:41 spiekey nope. i just change the device path from /dev/mapper/xxx  to /dev/sdb1. but the mount point is still the same
12:41 spiekey the deamon does not even come up. does gluster v start make sense at that point?
12:42 kanagaraj joined #gluster
12:42 glusterbot New news from newglusterbugs: [Bug 1109812] glusterd crash when detaching a node from existing cluster with peer detach command <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109812>
12:43 lalatenduM spiekey, yeah thats the root cause, when use a partition for gluster , xattrs set for bricks, when you reformatted the xattrs are gone, and gluster does not recognize it
12:44 spiekey oh, okay
12:44 spiekey how can i fix this now?
12:44 lalatenduM spiekey, for a working brick run "getfattr -d -m . <brick path>" you can see the xattrs
12:48 sroy joined #gluster
12:49 lalatenduM spiekey, or "getfattr -d -ehex -m . <brickpath>"
12:50 VerboEse joined #gluster
12:50 spiekey well, yes. whats that info useful for?
12:52 VerboEse joined #gluster
12:53 lalatenduM spiekey, the brick you have reformatted , if you can set the "rusted.glusterfs.volume-id" xattr to the previous value , it should fix the issue
12:53 lalatenduM s/rusted.glusterfs.volume-id/​trusted.glusterfs.volume-id/
12:53 glusterbot What lalatenduM meant to say was: spiekey, the brick you have reformatted , if you can set the "trusted.glusterfs.volume-id" xattr to the previous value , it should fix the issue
12:55 julim joined #gluster
12:55 spiekey lalatenduM: http://fpaste.org/110132/14029232/ they match
12:55 glusterbot Title: #110132 Fedora Project Pastebin (at fpaste.org)
12:56 jag3773 joined #gluster
12:57 lalatenduM spiekey, hmm interesting
12:59 spiekey ah!
13:00 coredump joined #gluster
13:00 spiekey in /var/lib/glusterd/peers/de2a51​5f-c728-449d-b91c-d80cabe0539f there was the hostname not written in fqdn.
13:00 spiekey just the short form
13:00 spiekey how could this happen?
13:00 lalatenduM not sure
13:01 primechuck joined #gluster
13:02 spiekey lalatenduM: thanks anyway
13:02 lalatenduM spiekey, the logs in http://fpaste.org/110124/02921524/ are from node01 or node02?
13:02 glusterbot Title: #110124 Fedora Project Pastebin (at fpaste.org)
13:02 spiekey node02
13:03 lalatenduM you are running command to start the volume in node 02 or 01?
13:04 B21956 joined #gluster
13:05 spiekey node02
13:05 spiekey node01 seemed to work all the time
13:07 lalatenduM spiekey, you can try running the command from node 1 with debug enabled, the log file might give some extra info
13:07 ctria joined #gluster
13:14 shyam joined #gluster
13:14 brad_mssw joined #gluster
13:15 bennyturns joined #gluster
13:15 kshlm joined #gluster
13:20 plarsen joined #gluster
13:29 kkeithley If you're still using GlusterFS-3.4.x, GlusterFS-3.4.4 RPMs for el5-7 (RHEL, CentOS, etc.) and Fedora (19, 20, 21/rawhide), are  now available in YUM repos at http://download.gluster.org/pu​b/gluster/glusterfs/3.4/LATEST
13:29 glusterbot Title: Index of /pub/gluster/glusterfs/3.4/LATEST (at download.gluster.org)
13:29 kkeithley There are also RPMs available for Pidora 20, SLES 11sp3 and OpenSuSE 13.1.
13:30 kkeithley Debian and Ubuntu DPKGs should also be appearing soon.
13:31 ikeakayke joined #gluster
13:34 harish joined #gluster
13:35 ikeakayke hi guys, first time irchatting, can anyone shine some light on what i'm doing wrong with gluster and vmware esxi 5.5? my setup is 2 esxi servers and another two with virtual ip sharing with gluster.
13:35 ikeakayke mounted datastore on virtual ip using gluster nfs
13:36 ikeakayke everything works ok regarding start/stop of virtual machines, but everything hangs and stall when activating HA and FT on machines
13:39 ikeakayke i think it has something to do with locks
13:39 ikeakayke the partitition on which gluster bricks are is xfs
13:40 kanagaraj joined #gluster
13:55 jiffe98 so I have 2 4 node 2 replica setup and I noticed nodes one and two have different disk usage sizes while 3 and 4 have the same
13:55 hagarth joined #gluster
13:57 jmarley joined #gluster
13:57 jmarley joined #gluster
13:58 tryggvil joined #gluster
13:59 daMaestro joined #gluster
14:02 davinder13 joined #gluster
14:03 bene2 joined #gluster
14:03 jobewan joined #gluster
14:06 brad_mssw JoeJulian: I noticed you were replying to a reported issue on the mailing list back in April, but cannot find if there was a resolution to the issue.  I was hoping you'd happen to know.  The thread in question is this one: http://supercolony.gluster.org/pipermai​l/gluster-users/2014-April/039985.html
14:06 glusterbot Title: [Gluster-users] libgfapi failover problem on replica bricks (at supercolony.gluster.org)
14:06 rotbeard joined #gluster
14:06 brad_mssw I appear to have produced this same issue when testing proxmox ve 3.2 and glusterfs 3.4.2
14:08 Pupeno semiosis: Sorry to bother you with a direct message, but do you know whether this bug https://bugs.launchpad.net/ubuntu/+sou​rce/glusterfs/+bug/876648?comments=all is present in GlusterFS 3.5?
14:08 glusterbot Title: Bug #876648 “Unable to mount local glusterfs volume at boot” : Bugs : “glusterfs” package : Ubuntu (at bugs.launchpad.net)
14:12 glusterbot New news from newglusterbugs: [Bug 1101942] Unable to peer probe 2nd node on distributed volume (3.5.1-0.1.beta1) <https://bugzilla.redhat.co​m/show_bug.cgi?id=1101942>
14:13 jbautista|brb joined #gluster
14:16 itisravi joined #gluster
14:19 lmickh joined #gluster
14:20 jbd1 joined #gluster
14:30 bene2 joined #gluster
14:44 tdasilva joined #gluster
14:45 lpabon joined #gluster
14:46 jiffe98 is gluster looking to change its method of replication?
14:52 koguma joined #gluster
14:55 primechuck joined #gluster
14:56 jcsp joined #gluster
14:58 ikeakayke joined #gluster
15:01 vpshastry joined #gluster
15:04 jag3773 joined #gluster
15:05 jbrooks joined #gluster
15:13 glusterbot New news from newglusterbugs: [Bug 1109889] [SNAPSHOT] USS: if snapd crashes and restarts or killed by SIGKILL and restarts, client cannot communicate with it <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109889>
15:22 jag3773 joined #gluster
15:23 bennyturns joined #gluster
15:40 bala joined #gluster
15:41 sjoeboo joined #gluster
15:41 _dist joined #gluster
15:42 brad_mssw I'm seeing some really strange behavior with gluster 3.4.2
15:42 brad_mssw I'm using it to store vm images, with replica 2, across 3 servers, each with 2 bricks
15:43 _dist what's the beahviour brad? I'm doing the exact same thing
15:43 brad_mssw i'm testing with proxmox 3.2
15:43 brad_mssw and what happens is inside the vm, the disk gets remounted as readonly
15:43 brad_mssw and if I forcibly stop the vm and start it again, I get a kernel panic during boot
15:44 brad_mssw what's really strange, is that kernel panic during boot goes away once that downed brick comes back online
15:44 brad_mssw so it seems like it is somehow depending on that downed brick
15:44 _dist can you dpaste your gluster volume info and give some details about your vm guest setup?
15:45 _dist wait, downed brick? need info the circumstances for that too :)
15:45 brad_mssw http://dpaste.com/10SZK76
15:45 glusterbot Title: dpaste: 10SZK76: Gluster volume info, by brad (at dpaste.com)
15:46 brad_mssw VM is configured to use virtio, and tried with either qcow2 or raw, both exhibit the same behavior
15:46 _dist can you describe the steps that lead up to the kernel panic or ro remount?
15:46 brad_mssw and by downed brick, I just reboot a server
15:47 _dist I was concerned about the time it might take libgfapi to change its' link, though in theory it shouldn't need to based on what I see in netstat. So I made gluster storage locations esrver :localhost instead of picking one in the proxmox setup
15:49 brad_mssw Ok, basically, my procedure is 1) set up proxmox ve 3.2 cluster with glusterfs ... 2) Create 3 identical VMs using virtio on the gluster volume one on each vm   3) Act like you are performing server maintenance   3a) migrate vms from node1 to node 2  3b) reboot node 1  3c) wait for gluster volume heal vms info to state nothing is left to be done   4) Repeat step 3 for each server  5) eventually one of the VMs will present with a message that / was remo
15:49 brad_mssw unted as ro, this usually happens after the second node is rebooted
15:50 brad_mssw I'm also not the only one to reproduce this: http://supercolony.gluster.org/pipermai​l/gluster-users/2014-April/039985.html
15:50 glusterbot Title: [Gluster-users] libgfapi failover problem on replica bricks (at supercolony.gluster.org)
15:50 brad_mssw but I haven't seen a resolution posted in that thread
15:51 _dist brad_mssw: your vms heal? Meaning that you don't see _any_ of them in gluster volume heal vol info ?
15:52 brad_mssw meaning that in  'gluster volume heal vms info' I see all 6 bricks, and each have a 'number of entries: 0'
15:52 brad_mssw which I assume when it goes into that state, that means everything is healed/healthy
15:52 brad_mssw (I don't see a better way to tell)
15:53 _dist (on a side note, that's awesome, and I can never get that myself), which is very interesting.
15:53 calum_ joined #gluster
15:54 _dist My best guess would be that for some reason when you take a brick down qemu libgfapi isn't smart enough to deal with it, to prove this you could see via netstat if it switches to a different IP, to avoid that concern entirely I configured my clients to "localhost"
15:54 _dist becaues in your config (same as mine) you'd never bring down the hypervisor that's running it, and the localhost glusterfs server
15:54 _dist because*
15:56 brad_mssw well, here's the even more interesting part of it all though ... I stop the vm, so the kvm process is down.  Then start it again, and things still aren't 'healthy', I get that kernel panic and right before it some message about an error in the root lv of the vm/guest.
15:56 jag3773 joined #gluster
15:56 brad_mssw but as soon as that rebooted server comes back up, I _can_ start it properly
15:56 brad_mssw it's like it has _some_ data, but it's not fully healthy data or something
15:57 brad_mssw so either the heal status is bad even though it reports as good, or there is some quorum related issue
15:58 _dist if my guess is correct you took down the one named for the /mnt/pve location?
15:58 _dist or does it matter which one you take down?
15:58 brad_mssw I'm not sure if it matters, I have gluster set up to use 'localhost'
15:59 brad_mssw I'm not naming one host the master or anything
15:59 brad_mssw libgfapi should handle that properly anyhow, even for fuse mounts since it is used underneath
16:00 brad_mssw I'm really not having much luck locating any sort of client-side libgfapi logs
16:00 SFLimey joined #gluster
16:01 _dist I agree that it should, and it doesn't sound like that's your issue (if you're using localhost)
16:02 _dist However, I'm also puzzled about you seeing 0 entries in the heal stat, for me all my vms always show in that stat if I do a watch on it, I've been working with JoeJulian to resolve it, I think it's next on his list.
16:02 _dist But, based on your experience it sounds like that problem might not be 100% reproducable
16:02 rotbeard joined #gluster
16:02 brad_mssw _dist: well, I can reproduce it on my system 100% of the time
16:02 dusmant joined #gluster
16:02 brad_mssw _dist: and that mailing list entry I posted could too
16:03 brad_mssw _dist: I just don't know if it is some config thing somewhere I need to tweak or what
16:06 _dist oh, your test is a failure of gluster test, where you don't gracefully stop the service? I believe that is what the linked post was about
16:07 davinder13 joined #gluster
16:07 brad_mssw well, I haven't tried directly messing with the service, I'm just issuing a 'reboot' ... in the entire chain, I think that poster was originally doing what I'm doing ... then he shortened it in a test environment
16:07 primechuck joined #gluster
16:08 brad_mssw but let me go ahead and kill it gracefully and see if it happens or not
16:08 _dist there must be something different about our setups, I perform reboots regularily after migration and if what you're describing ever happened to me, it would be a _huge_ deal
16:08 rjoseph joined #gluster
16:09 brad_mssw proxmox defaults to ide for disk, are you sure you're using virtio like me?
16:09 _dist the only thing different about your volume settings than mine is the insecure on section, I've found that with proxmox that's not required because it runs things as root
16:10 _dist yeap, every disk is virtio, cache=none, almost all except one are qcow2, the other is raw
16:11 vpshastry joined #gluster
16:11 brad_mssw heh, just trying to   service glusterfs-server stop   doesn't actually stop glusterfs
16:12 brad_mssw if I kill the glusterfs and glusterfsd processes, after about 10s I get 'Aborting journal on device dm-0-8.  Buffer I/O error on device dm-0, logical block 3702784
16:12 _dist oh I see why your test is different than anything I would ever actually do
16:12 brad_mssw and it remounts the fs as readonly
16:12 _dist and I suspect I'd have the same issue because it's this -> http://supercolony.gluster.org/pipermai​l/gluster-users/2014-April/039987.html that causes the problem
16:12 glusterbot Title: [Gluster-users] libgfapi failover problem on replica bricks (at supercolony.gluster.org)
16:13 _dist I have always migrated my machines to a new node, before taking down either node. And I would always do the same before doing that again
16:14 brad_mssw I just make sure the vms are not on the node that is going to be shut down
16:14 _dist So if I migrate all VMs from host 1 to 2, take 1 down, wait for it to heal, keep vms on 2 but kill 2's gluster
16:15 _dist I've never done that, but that is the right way to see if libgfapi re-negotiates properly while qemu-kvm is running.
16:15 _dist We go out of our way to make sure if a VM is running on proxmox node 1, it's using libgfapi pointed to that local glusterfs
16:16 _dist (which might explain why I've never had this problem)
16:17 brad_mssw _dist: do you use replica 2 or replica 3?
16:18 brad_mssw I can't get this vm to start up anywhere right now when this node is down, it doesn't make any sense ... the data appears to be available
16:18 _dist 2 right now, plan to move to 3 shortly. If you stop and start the vm after it crashes, does it work or still fail?
16:18 brad_mssw still fails _until_ I bring up that other gluster node
16:18 brad_mssw then it works
16:19 brad_mssw I've tried starting it on the other 2 vm servers, neither works, so it isn't isolated to one host
16:19 _dist if you look at your kvm command what does the disk section say?
16:19 SFLimey joined #gluster
16:19 _dist (in ps, while its' running but failing)
16:19 brad_mssw file=gluster://localhost/vms/images/100/​vm-100-disk-1.qcow2,if=none,id=drive-vir​tio0,format=qcow2,aio=threads,cache=none
16:20 brad_mssw (I made the aio=threads change as a test as it was suggested on the proxmox forum, it is normally aio=native .. neither work)
16:20 _dist ... the only thing I could guess is that your localhost brick doesn't think it has the whole file?
16:20 _dist what is your FS that supports gluster?
16:21 brad_mssw the fs on the bricks is XFS
16:22 _dist At this point, assuming I've understood everything correctly I'm just as confused as you are why that kvm machine won't boot.
16:23 _dist I'm not a gluster dev, never even looked at the code beyond getting it to run on BSD. I only have practical experience using it
16:23 cfeller kkeithley: thanks for the Fedora and EL 3.4.4 rpms!
16:23 _dist My FS is zfs, and I've run it on ext4, but xfs completely supports xattrs so I can't see why it'd be different.
16:24 JoeJulian brad_mssw: I've tried duplicating that bug with a fuse mount as I can't see any reason why it should succeed where a libgfapi doesn't since they use the same translators. No luck though.
16:25 brad_mssw JoeJulian: were you able to replicate it with libgfapi though?
16:25 brad_mssw (not using fuse) ?
16:25 JoeJulian Not yet. Our staging environment's qemu is too old.
16:26 brad_mssw ah
16:26 kkeithley cfeller: yw
16:26 brad_mssw actually, right now with fuse, i tried to rm the offending image, and it says it is readonly
16:26 JoeJulian ... and, of course, it's a new job and they've got broken stuff I need to fix. :D
16:26 brad_mssw but the mount itself shows as rw
16:26 _dist JoeJulian: I can attempt to replicate this if it'll help, I'll setup a nested kvm cluster
16:27 _dist brad_mssw: did you setup quorum on gluster? Maybe that's why you can't read/write to /mnt/pve/name ?
16:27 brad_mssw _dist: I _could_ be missing something there since I'm new to gluster ... so where should I look
16:28 brad_mssw _dist: http://dpaste.com/10SZK76 showed cluster.server-quorum-type: server and cluster.quorum-type: auto
16:28 glusterbot Title: dpaste: 10SZK76: Gluster volume info, by brad (at dpaste.com)
16:28 brad_mssw which is what I thought was needed ... but if there's more, that could definitely be the issue
16:29 _dist I haven't set any quorum parms on my volumes yet because I don't have a 3rd brick yet, I know the wrong sequence of events could give me a split-brain because of that.
16:30 brad_mssw ok, well, that's a major difference in our setups then
16:30 kmai007 joined #gluster
16:30 _dist I'm not familiar with doing the quorum setup for gluster, so I can't say. But if with one node you only RO on gluster, that would explain not being able to boot a vm
16:30 _dist bbiab lunch
16:34 brad_mssw JoeJulian: any idea how to tell why a file may be marked as readonly?
16:35 JoeJulian quorum
16:35 JoeJulian Or the brick is read only
16:35 brad_mssw 4 of 6 bricks are currently online
16:36 brad_mssw (doing 2 bricks per server, so 3 total servers, 1 server is offline)
16:36 brad_mssw verified the underlying file systems are all 'rw'
16:36 brad_mssw where do I check to make sure glusterfs is doing proper quorum ?
16:36 JoeJulian @pateinfo
16:36 glusterbot JoeJulian: I do not know about 'pateinfo', but I do know about these similar topics: 'pasteinfo', 'pathinfo'
16:37 Matthaeus joined #gluster
16:37 JoeJulian hehe, yes. Tell us about your hair... :/
16:37 JoeJulian @pasteinfo
16:37 glusterbot JoeJulian: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
16:37 brad_mssw JoeJulian: http://dpaste.com/10SZK76
16:37 glusterbot Title: dpaste: 10SZK76: Gluster volume info, by brad (at dpaste.com)
16:38 keytab joined #gluster
16:38 kanagaraj joined #gluster
16:39 JoeJulian ok, so you're using server quorum. I thought that was supposed to shut down gluster services on a server that loses quorum.
16:39 kanagaraj joined #gluster
16:39 brad_mssw no idea, just some wiki said to do that to enable quorum
16:39 brad_mssw should I use something else?
16:45 jrcresawn joined #gluster
16:46 ProT-0-TypE joined #gluster
16:47 JoeJulian brad_mssw: https://github.com/gluster/glusterfs/bl​ob/master/doc/features/server-quorum.md
16:47 glusterbot Title: glusterfs/doc/features/server-quorum.md at master · gluster/glusterfs · GitHub (at github.com)
16:47 JoeJulian brad_mssw: Since that's not fully configured and there are no defaults, I'm going to assume that's doing nothing.
16:48 JoeJulian Since cluster.quorum-type is auto, it /should/ only go ro if less than bricks / 2 + 1 are up.
16:50 * JoeJulian is going to file a bug report for something completely unrelated
16:50 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
16:50 brad_mssw 4 bricks are up, so that should be fine
16:50 brad_mssw what do you suggest I change?
16:51 JoeJulian Mount the volume again someplace else and see if that's still ro. If it is, paste the new client log.
16:55 brad_mssw fyi, I was able to successfully delete a different file on the same mountpoint
16:55 brad_mssw trying to mount elsewhere and delete the file that is giving issues ...
16:56 brad_mssw ok, so same error on a different system trying to delete that file, says it is read-only
16:56 brad_mssw log does state 'failing unlink due to lack of quorum'
16:57 sjm joined #gluster
16:57 brad_mssw need to figure out why it thinks quorum isn't there then
16:57 sjm left #gluster
16:57 JoeJulian volume status maybe?
16:58 nage joined #gluster
16:58 JordanHackworth joined #gluster
16:58 brad_mssw JoeJulian: http://dpaste.com/17X99N2
16:58 glusterbot Title: dpaste: 17X99N2: volume status, by brad (at dpaste.com)
16:59 _dist JoeJulian: I find it interesting thta brad doesn't have my healing issue, I wonder if it's related to quorum, or xfs. But I'm pretty sure I had it on xfs.
17:00 brad_mssw is there a way to query gluster quorum?
17:02 _dist I was wondering if anyone knew what volume options can be used to speed up small writes?
17:02 JoeJulian Not doing them.
17:03 JordanHackworth joined #gluster
17:03 JoeJulian The smaller the write, the bigger the cost of tcp headers and latency.
17:04 JoeJulian If your file is open, a small write shouldn't be that big of a deal. If you're closing the file between writes, disabling self-heal will spead that up.
17:04 JoeJulian spead?
17:04 _dist JoeJulian: is that why? on 1k writes I get about 80x slower than local disk, but large writes are 90% of local speed
17:04 * JoeJulian needs more coffee
17:05 kmai007 disabling self.heal is a per volume feature?
17:05 tyrok_laptop joined #gluster
17:08 JoeJulian kmai007: yes
17:08 tyrok_laptop In the output of "gluster volume heal [volumename] info heal-failed", I get 1024 items with paths like <gfid:[a bunch of hex digits]> instead of a path.  What causes this disconnect between apparently un-healable items and their paths?
17:08 tryggvil joined #gluster
17:11 brad_mssw JoeJulian: is quorum being handled on a per-brick level?
17:11 brad_mssw JoeJulian: as I just set   quorum-type fixed and quorum-count 1   and it works
17:12 dtrainor joined #gluster
17:12 brad_mssw JoeJulian: which it is true, _one_ of the 2 bricks in the replica is down
17:12 brad_mssw JoeJulian: but overall, the set of servers still has quorum
17:13 JoeJulian tyrok_laptop: Self-heal queues are stored in .glusterfs/indices/xattrop as links to the gfid file under .glusterfs. If the client (probably glustershd) hasn't touched the file with that gfid, it cannot resolve the gfid to a filename and will just report the gfid. You can sometimes overcome that by doing a "heal $vol full" and waiting.
17:13 brad_mssw JoeJulian: and cluster.quorum-type and cluster.quorum-count are what controls client-side behavior, right?  do I need that if server-side is enabled?
17:13 lpabon joined #gluster
17:14 JoeJulian Right, those are client side. server-side will kill the brick if it can't find a server quorum.
17:14 tyrok_laptop JoeJulian: Cool.  Crawling files with find would work then, too, correct?
17:14 brad_mssw it sounds like the client-side quorum logic is wrong
17:14 brad_mssw unless it really is intended to be per-brick quorum
17:14 JoeJulian tyrok_laptop: I believe it should, yes.
17:15 brad_mssw in which case with a replica 2, loss of a server will always result in this behavior
17:15 tyrok_laptop JoeJulian: Cool.  Thanks for the help!
17:15 JoeJulian brad_mssw: It's probably per replica set, which would make sense.
17:15 JoeJulian @gfid
17:15 glusterbot JoeJulian: The gfid is a uuid that's assigned to represent a unique inode that can be identical across replicas. It's stored in extended attributes and used in the .glusterfs tree. See http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/ and http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
17:16 JoeJulian @gfid resolver
17:16 glusterbot JoeJulian: https://gist.github.com/4392640
17:16 JoeJulian tyrok_laptop: ^
17:16 cfeller I have four gluster servers that will have to move to a different subnet (I work at a university). Theoretically, it should be a simple as updating DNS, but three of the four nodes in the cluster somehow have the address of the first node hard coded as an IP.  I'm not sure how this happened, as I created this system specifying hostnames and not IPs.
17:16 tyrok_laptop JoeJulian: Awesome.  Thanks!
17:17 cfeller Here is what I'm looking at: http://ur1.ca/hjm2x
17:17 glusterbot Title: #110210 Fedora Project Pastebin (at ur1.ca)
17:17 cfeller One, how to fix this, two what is the best way to change the IP address of gluster servers?
17:17 JoeJulian cfeller: You just missed one step in setting up your peers. As long as your volumes are defined with hostnames you can fix that first peer by just probing it by name from any other peer.
17:18 cfeller JoeJulian: ah, that works!
17:19 cfeller JoeJulian: any other caveats changing IP addresses when I move the servers to a different subnet?
17:19 tyrok_laptop JoeJulian: That gist is a great help.
17:19 JoeJulian Shouldn't be.
17:19 cfeller JoeJulian: OK, thanks!
17:19 JoeJulian @ semiosis++
17:19 glusterbot JoeJulian: semiosis's karma is now 2
17:19 semiosis :O
17:19 sputnik13 joined #gluster
17:27 brad_mssw JoeJulian: ok, what are the implications of running only server-side quorum?  is split-brain possible?
17:27 brad_mssw JoeJulian: I would think server-side quorum would prevent split-brain, but maybe my understanding of glusterfs isn't right
17:29 ProT-0-TypE joined #gluster
17:31 julim joined #gluster
17:31 _dist Is there any way to a summary of client level info, like write block size, utilization of the server, etc? Maybe a way I can log it, at the gluster level
17:34 JoeJulian brad_mssw: Yes, only server-side should be able to prevent split-brain.
17:36 brad_mssw JoeJulian: it looks like server-side quorum was a fairly recent addition, is it meant to replace client-side quorum?  meaning is there any purpose at all to client-side quorum if server-side quorum is enabled?
17:39 julim joined #gluster
17:39 JoeJulian They serve different use cases.
17:40 kmai007 does anybody have any doc. on how to setup separation of the gluster management traffic from the gluster file traffic?
17:40 JoeJulian server-side prevents split brain at the loss of accessibility to data. client-side still allows ro access.
17:40 JoeJulian kmai007: Not really. Just have the hostnames resolve differently on the servers than it does on the clients.
17:41 JoeJulian aka split-horizon dns
17:42 tyrok_laptop JoeJulian: Okay, so looking up one of those GFID paths, I am able to stat and read the heal-failed files just fine.  However, the path is still not showing up in gluster volume heal info heal-failed, and its GFID is still on the list.  Any ideas?
17:43 JoeJulian tyrok_laptop: those are logs. Unless they show up with a recent timestamp, they may have already been healed. If the timestamp is recent, check the glustershd.log file.
17:43 zaitcev joined #gluster
17:43 kmai007 tyrok_laptop: if i recall the gluster volume heal info <options> is a static report, it doesn't change until glusterd is restarted
17:43 glusterbot New news from newglusterbugs: [Bug 1109950] [feature] "gluster volume status" could report version <https://bugzilla.redhat.co​m/show_bug.cgi?id=1109950>
17:43 tyrok_laptop JoeJulian, kmai007: Ah, gotcha.
17:44 tyrok_laptop Timestamps on these are in the future.  They are newer than system date.
17:44 kmai007 in the future...sounds like a fortune teller
17:45 JoeJulian tyrok_laptop: gmt
17:45 kmai007 oh yeh thats right
17:45 tyrok_laptop K
17:45 kmai007 greenwich
17:45 JoeJulian or utc, depending on how old you are.
17:46 tyrok_laptop So maybe I'd be better off doing a restart and then doing a full heal just to make sure it only displays current problems?
17:47 JoeJulian Heh, isn't human nature fascinating.
17:48 vpshastry joined #gluster
17:52 hchiramm__ joined #gluster
17:53 tyrok_laptop JoeJulian: I take it that's not the ideal way to try and figure out if there are current problems which need to be solved?
17:53 LebedevRI hello. Debian sid, gluster 3.5. i have an issue: i had to do following - http://gluster.org/community/documen​tation/index.php/Gluster_3.4:_Brick_​Restoration_-_Replace_Crashed_Server i succeeded, but i can not run heal - "Status: self-heal-daemon is not running on ..." what should i do?
17:53 glusterbot Title: Gluster 3.4: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at gluster.org)
17:54 JoeJulian tyrok_laptop: It's not any more efficient than looking at timestamps, but everyone does it.
17:55 JoeJulian LebedevRI: restart glusterfs-server on the host that it claims is missing the self-heal daemon and see if that solves it.
17:55 tyrok_laptop JoeJulian: Gotcha.  Thanks again for the help!
17:56 lpabon_ joined #gluster
17:59 lpabon joined #gluster
18:00 StarBeast joined #gluster
18:01 Matthaeus1 joined #gluster
18:02 haomaiwa_ joined #gluster
18:02 lpabon joined #gluster
18:06 LebedevRI no, that did not help. i restarted it on both servers, but the output is still the same
18:09 JoeJulian @pastestatus
18:09 glusterbot JoeJulian: Please paste the output of gluster peer status from more than one server to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
18:09 JoeJulian LebedevRI: ^
18:11 LebedevRI https://dpaste.de/YWsp
18:11 glusterbot Title: dpaste.de: Snippet #271742 (at dpaste.de)
18:13 hchiramm__ joined #gluster
18:23 ricky-ticky1 joined #gluster
18:25 kmai007 has anybody had experience using nfs-ganesha on glusterfs?
18:27 ramteid joined #gluster
18:29 brad_mssw when trying to get heal info, I am getting "Another transaction is in progress. Please try again after sometime."
18:30 brad_mssw any idea what causes that?
18:30 brad_mssw that happens on all nodes
18:31 JoeJulian LebedevRI: Gah, I forgot that was peer status. I'm looking for volume status.
18:32 LebedevRI JoeJulian: https://dpaste.de/1rvA
18:32 glusterbot Title: dpaste.de: Snippet #271745 (at dpaste.de)
18:32 JoeJulian brad_mssw: It thins that some other cli command is still being processed. This causes a transaction lock on glusterd. The only way I know to solve that is to restart all glusterd.
18:32 brad_mssw I've got a glusterd process eating up 100% CPU
18:32 brad_mssw and eating 8.8G ram so far
18:32 brad_mssw and growing
18:32 JoeJulian nefty
18:32 brad_mssw 9.8G
18:32 JoeJulian nifty even
18:33 brad_mssw 10.8G
18:33 brad_mssw niiice
18:33 JoeJulian just kill it
18:34 brad_mssw JoeJulian: you ever seen that behavior?
18:34 JoeJulian LebedevRI: Nice... no self-heal daemons at all.
18:35 brad_mssw i killed it on all nodes, seems ok now
18:35 JoeJulian brad_mssw: nope. I don't suppose you know how you did that?
18:35 JoeJulian LebedevRI: Let's paste up a glustershd.log
18:36 brad_mssw JoeJulian: I was rebooting nodes and waiting for the volume heal info to return normal
18:36 brad_mssw JoeJulian: just basic QA
18:36 brad_mssw I had   'watch gluster volume heal vms info'   running on one console
18:37 brad_mssw so may have been a race condition of some sort
18:37 brad_mssw that triggered the behavior
18:37 JoeJulian file a bug
18:37 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
18:37 brad_mssw well, I'm going to see if I can reproduce it before I do that
18:38 JoeJulian ok
18:38 JoeJulian If you can't, still file it. I've seen a2 figure out a race with less info to go on.
18:39 LebedevRI JoeJulian: nothing to paste - they are empty on both servers
18:39 JoeJulian df?
18:40 JoeJulian maybe
18:41 LebedevRI_ joined #gluster
18:42 LebedevRI_ JoeJulian: https://dpaste.de/B8ZT
18:42 glusterbot Title: dpaste.de: Snippet #271746 (at dpaste.de)
18:47 bchilds joined #gluster
18:49 mkzero joined #gluster
19:08 plarsen joined #gluster
19:23 shapemaker joined #gluster
19:24 pdrakeweb joined #gluster
19:27 hchiramm__ joined #gluster
19:30 Pupeno_ joined #gluster
19:47 qdk joined #gluster
19:47 ctria joined #gluster
19:59 LebedevRI JoeJulian: any more ideas?
20:08 brad_mssw there's still something up with libgfapi access, rebooting a brick out from under a running vm can still sometimes cause fs issues
20:09 brad_mssw even though there is another brick that is part of the replica still online
20:10 LebedevRI joined #gluster
20:12 dblack joined #gluster
20:21 StarBeast joined #gluster
20:22 P0w3r3d joined #gluster
20:27 dtrainor joined #gluster
20:27 dtrainor 'afternoon!  So I'm still working with this single-peer Gluster setup and trying to figure out how to rebuild a brick after I've physically replaced it.  I see a lot of information about rebuilding entire peers or adding peers, not so much on a one-for-one replacement of a *brick* itself.  Anyone got a minute?
20:33 JoeJulian It's a shame that there's no utility to just set the brick id from the cli...
20:33 JoeJulian dtrainor: so basically, just replace the disk and mount it.
20:34 JoeJulian ... create any directory tree you need...
20:34 dtrainor right, it's mounted right where the old one was
20:34 dtrainor gave it a new fs label and it's there
20:35 dtrainor 'gluster volume status' says the brick is not Online though....
20:35 JoeJulian pick another brick and run "getfattr -n trusted.glusterfs.volume-id $brick_path"
20:35 JoeJulian pick another brick and run "getfattr -n trusted.glusterfs.volume-id -d -e hex $brick_path"
20:35 JoeJulian better yet... ^
20:35 dtrainor any existing brick?
20:35 JoeJulian yes
20:36 dtrainor ok
20:36 dtrainor I see:  trusted.glusterfs.volume-id=0xd​29547020f1943c79b470471970d9019
20:36 JoeJulian ... then: setfattr -n trusted.glusterfs.volume-id -v $value_from_above $new_brick_path
20:36 JoeJulian ... finally: gluster volume start $volume_name force
20:36 dtrainor what if it's already started?
20:37 JoeJulian That brick didn't start because of the missing volume-id (a feature).
20:37 JoeJulian So start...force will just force the missing brick to start.
20:37 dtrainor got it.  1s
20:38 dtrainor soo.... we took the volume uuid that bricks need to associate with the volume, assigned that to the brick, and start forced the volume to re-scan for new bricks with this same ID?
20:39 andreask joined #gluster
20:39 sjm joined #gluster
20:41 dtrainor that's cool.  how long will it take for that data to get put back on that brick?
20:41 dtrainor it was a Distributed-Replicate, 2x2=4
20:42 JoeJulian Depends on load, amount of data, network speed, phase of the moon...
20:44 glusterbot New news from newglusterbugs: [Bug 1110018] Not able to set deny acls on samba shares <https://bugzilla.redhat.co​m/show_bug.cgi?id=1110018>
20:50 glusterbot New news from resolvedglusterbugs: [Bug 870256] Samba "store dos attributes" feature doesn't work with GlusterFS. <https://bugzilla.redhat.com/show_bug.cgi?id=870256>
21:00 Matthaeus joined #gluster
21:03 Mo__ joined #gluster
21:12 oxidane joined #gluster
21:12 ron-slc joined #gluster
21:12 Ramereth joined #gluster
21:17 diegows joined #gluster
21:21 Matthaeus joined #gluster
21:27 tyrok_laptop left #gluster
21:27 sjm left #gluster
21:27 diegows joined #gluster
21:40 Matthaeus joined #gluster
21:49 diegows joined #gluster
22:00 julim joined #gluster
22:13 sputnik13 joined #gluster
22:16 japuzzo joined #gluster
22:17 JoeJulian LebedevRI: Next, I'd check etc-glusterfs-glusterd.vol.log to see if it's having any problems spawning the self-heal daemon, glustershd. (don't paste it unless you see something you need interpreted. I trust you to make your own analysis.)
22:39 LebedevRI i ended up completely removing all gluster configs except .glusterfs in bricks and recreating volumes; seems to helper
22:41 JoeJulian cool
22:51 elico are there any open\free appliances based on glusterfs?
23:05 atrius` joined #gluster
23:06 dtrainor JoeJulian, I see data starting to pour in to the brick, thanks again for your help
23:06 dtrainor Is there a status or a % that I can watch?
23:06 JoeJulian You're welcome :)
23:06 JoeJulian Unfortunately, no.
23:08 tryggvil joined #gluster
23:09 JoeJulian I generally use df and just watch for them to converge.
23:09 dtrainor ok
23:10 dtrainor that's gonna take a while
23:10 dtrainor 2x2=4, each brick is 2tb, total volume size 3.7T
23:11 JoeJulian Isn't that cute... ;)
23:11 dtrainor haha
23:11 JoeJulian I can say that now that I don't have the smallest volume on the planet.
23:11 dtrainor you're welcome
23:11 dtrainor see, we help each other
23:11 dtrainor i got sick of half-assing disk space at home soooo that's wha thappened
23:16 Matthaeus JoeJulian: mine is smaller.
23:16 sputnik13 joined #gluster
23:19 tdasilva left #gluster
23:21 JoeJulian Matthaeus: Can you believe I stayed away from "That's what she said"?
23:24 Matthaeus JoeJulian: But did you?  Did you really?
23:24 JoeJulian hehe
23:25 Matthaeus I'm going to be doing a 20TB ZFS box pretty soon, but my production gluster volume is around 600 gigabytes.
23:26 JoeJulian Where I'm at now, one of my bricks is 48TB.
23:27 JoeJulian ... and don't try and tell me it's not the size that counts...
23:31 MacWinner joined #gluster
23:31 Matthaeus 48 TB usable or raw?
23:31 JoeJulian usable
23:45 elico JoeJulian: with ZFS or others?
23:46 JoeJulian elico: standard xfs
23:46 elico xfs ontop of raw partition ? lvm?
23:47 elico (just wondering)
23:47 JoeJulian xfs on raid 6 raw
23:47 elico ok seems reasonable.
23:47 JoeJulian 4 bricks per server, 8 servers per rack, 15 racks per module
23:48 elico that's a lot...
23:49 elico Maye CocaCola can take usage of this huge amount of storage.. they are using IBM DB2 and they have lot's of data in it.
23:50 JoeJulian 11.5 replicated petabytes per module. Currently building 5 modules
23:50 elico ouch..lots :D
23:51 JoeJulian I've been told we're expecing to have 25 of those petabytes online by the end of the year.
23:51 * JoeJulian is practically giddy with his new toys.
23:52 elico What are they holding there? the DNA of every bacteria on the planet?
23:52 JoeJulian cloud provisioning.
23:53 JoeJulian You know how I get pissy with people for wanting glusterfs to be perfect at everything for everybody... well now I'm one of them.
23:53 sputnik13 joined #gluster
23:53 JoeJulian At least, though, I have the experience to analyze potential problems and submit actionable bug reports.
23:53 Pupeno joined #gluster
23:54 elico I wish them luck!! for me to even see 10TB on top of couple racks is amazing.
23:54 * JoeJulian sheepishly kicks the dirt...
23:54 Ark joined #gluster
23:54 JoeJulian my home media server is 8 TB...
23:54 elico hehe
23:56 elico Well I do have the disks here for 8TB but one of them is damaged WD black which needs to be replaced but takes lots of time to backup and the others are 1tb*2 2tb*4 and everybody here is happy that they have 2tb of storage on their PC right there at the distance of a click :D
23:57 elico The only issue is that it takes electricity and somebody needs to pay for that :
23:57 elico :\

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary