Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 ninkotech__ joined #gluster
00:01 ktosiek joined #gluster
00:08 cp0k Interesting, after upgrading Gluster to 3.4.2, on some boxes the /etc/init.d/glusterd start up script worked just fine
00:08 cp0k on other boxes I needed to comment out
00:08 cp0k #command_args="-N"
00:13 ninkotech__ joined #gluster
00:19 tdasilva left #gluster
00:22 hagarth joined #gluster
00:22 cp0k semiosis: you still around by any chance?
00:26 ninkotech__ joined #gluster
00:29 overclk joined #gluster
00:33 recidive joined #gluster
00:35 ninkotech__ joined #gluster
00:45 ninkotech_ joined #gluster
00:46 glusterbot New news from resolvedglusterbugs: [Bug 918052] Failed getxattr calls are throwing E level error in logs. <https://bugzilla.redhat.com/show_bug.cgi?id=918052>
00:54 vpshastry joined #gluster
00:56 ninkotech joined #gluster
00:56 ninkotech_ joined #gluster
00:57 ninkotech_ joined #gluster
01:00 spiekey_ joined #gluster
01:00 andreask joined #gluster
01:07 bala joined #gluster
01:07 ninkotech_ joined #gluster
01:09 chirino joined #gluster
01:10 rfortier1 joined #gluster
01:11 Elico joined #gluster
01:17 purpleidea @later tell primechuck https://gluster.org/pipermail/glust​er-users/2014-February/039039.html
01:17 glusterbot purpleidea: The operation succeeded.
01:18 purpleidea ^^^ my recent test showing HA. if anyone knows if there's a fix to avoid the 42 second delay before failover, i think that should be a new gluster feature :) <-- yes i know you can change the value of 42.
01:21 recidive joined #gluster
01:22 ninkotech joined #gluster
01:27 ninkotech joined #gluster
01:29 jporterfield joined #gluster
01:33 recidive joined #gluster
01:42 Matthaeus joined #gluster
01:54 vpshastry joined #gluster
01:59 shyam joined #gluster
02:08 diegows joined #gluster
02:16 shyam joined #gluster
02:16 codex joined #gluster
02:25 bharata-rao joined #gluster
02:25 harish joined #gluster
02:29 shyam joined #gluster
02:50 _dist joined #gluster
02:54 Guest70826 evening everyone, I know I keep asking the same question, but I'm really stuck on not knowing what is and isn't synced if using gluster for kvm (a great reason to use it). But I'm stuck.
02:55 Guest70826 oh nickserv wrecked me :) just a sec
02:55 Guest70826 left #gluster
02:57 dddfdf joined #gluster
02:57 dddfdf there we go, I was that guest a min ago
02:57 dddfdf left #gluster
03:05 ilbot3 joined #gluster
03:05 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
03:05 johnmark joined #gluster
03:09 _dist I've been thinking, if 99% of my self heal entries are actually out of sync replication of running vms (correct me i I'm wrong), maybe there's a way to limit what heal info shows to entries that are x seconds out of sync?
03:09 edong23 _dist: this seems similar to an issue smellis_ was having...
03:10 _dist edong23: What was that? (my situation: 2 node replicate volume, constatly healing if vms are powered on) vms are running through libgfapi
03:11 edong23 i dont know about libfap
03:11 edong23 but, yes to the rest
03:11 edong23 constantly healing
03:11 edong23 even with 20gig infinerband interconnect
03:11 edong23 and sometimes, during healing had performance issues
03:11 _dist edong23: was there a bug entry? setup issue? Right, I have 2x10GB on each side
03:12 edong23 um..
03:12 edong23 i am trying to get him in here
03:13 edong23 but, ultimately he tore down the gluster setup and did standaone hosts until he can sort it out
03:13 _dist edong23: cool, I've honestly never seen a different behaviour, and before I go live... oh got it
03:13 edong23 he thinks... his hosts are doing too much.  hosting VMs, network, and gluster...
03:13 edong23 _dist: he didnt have these issues in the lab though
03:14 edong23 he didnt see performance issues until live time
03:14 edong23 smellis_: goddamn it
03:14 edong23 get in here
03:14 _dist edong23: I doubt that's the case in my situation. Mine is a test setup, single vm does it 24 cores, 300GB of ram on each box. I'm patient no rush just glad someone else has the same issue
03:15 edong23 _dist: thats the thing... i asked him if he really thought it was the case... which ... he could mean the performance thing is caused by his lack of excess resources
03:16 edong23 the constantly healing thing could jsut be a symptom...
03:16 edong23 im not sure
03:16 edong23 i plan on having dedicated gluster storage
03:16 edong23 but with your 24 cores...
03:16 edong23 jesus dude
03:16 edong23 you must be government?
03:17 edong23 are those e5 or e7?
03:18 _dist edong23: I don't think it's a performance issue, probably an implemetation mistake on my end. If you build systems yourself you'd be surprised what you can do for just a thousand. e5s
03:18 edong23 _dist: i build my systems
03:18 edong23 but those cpus are like 2-3k each...
03:18 edong23 ill stick with my $200 quad cores..
03:19 _dist edong23: mostly super-micro stuff, bulk of $ in ram and processing yeah
03:19 edong23 thats fucking sick though dude
03:19 edong23 do me a favor,    enable the bootlogo in the kernel, boot it up with framebuffer and take a picture of the penguins
03:19 _dist edong23: yeah, I'll admit I'm pretty excited to get this thing going live :)
03:20 edong23 _dist: when i build new servers, itll be super-micro
03:20 edong23 my current ones are asus.they have been great though
03:20 edong23 rock solid
03:20 sticky_afk joined #gluster
03:20 shubhendu joined #gluster
03:21 stickyboy joined #gluster
03:21 _dist edong23: I was iffy on it in our first order to be honest, but everything I've had from them from case to accessory has been real quality stuff. Hot swap drive trays $5/each and it comes full (case for example). I've used most major pre-built brands, so far they don't compare (but I'm open, not close minded about it)
03:22 edong23 yeah, i have a few supermicro servers. they are great...
03:23 edong23 albeit, old
03:23 edong23 new stuff would be amazing
03:23 edong23 _dist: did you go 10gb ethernet?
03:24 _dist edong23: yeah, they have it on the board to begin with, then I bought a 2x10gb pcie card for each (now those honestly are pricey, like $500 each min).
03:25 edong23 damn
03:25 edong23 i could have sold you 3 brand new ones for 350 each
03:25 _dist really, what brand?
03:25 edong23 intel
03:26 edong23 i bought 3, tested them, and then went with 40gig infiniband
03:26 edong23 because of cost to speed ratio
03:26 edong23 and ill have 6 nodes participating
03:26 edong23 cause i dont have 24 cores
03:26 edong23 now im jsut sittin her with these cards like "welp..."
03:26 _dist I've found honestly the best I can get out of them is about 15gbps bonded with a sustained iperf, not 20
03:27 edong23 _dist: thats not bad... actually
03:27 edong23 round robin?
03:27 _dist ad for now
03:28 edong23 i cant believe you get that speed...
03:28 _dist jumbo frames didn't have that much of an impact, it was mostly txquelen that did it
03:28 kshlm joined #gluster
03:28 edong23 but... gluster is tcp
03:28 edong23 yeah?
03:28 edong23 so... ad should stick that on a single nic
03:28 edong23 ...
03:29 _dist edong23: honestly I don't at the FS level, and yeah gluster has overhead too, but I have the extra ports just don't want a network bottleneck
03:29 RameshN_ joined #gluster
03:29 RameshN joined #gluster
03:29 _dist and iftop shows about 2-3gbps on a cp to a gluster fuse mount
03:29 kanagaraj joined #gluster
03:30 edong23 _dist: yeah... it is latency that really matters, is what me and smellis_ came up wit
03:30 edong23 yeah
03:30 edong23 with infiniband, that will likely be higher
03:30 edong23 10gig over copper doesnt have phenominal low latency
03:30 edong23 it is good, but not fantastic
03:30 edong23 infiniband is made for that
03:30 edong23 and cheap
03:30 smellis_ yerp here
03:31 _dist yeap, I considered it, maybe an uneducated decision to go with ethernet.
03:31 edong23 finally, you dick
03:31 edong23 did you read the backlog?
03:31 edong23 <_dist> I've been thinking, if 99% of my self heal entries are actually out of sync replication of running vms (correct me i I'm wrong), maybe there's a way to limit what heal info shows to entries that are x seconds out of sync?
03:31 _dist smellis: edong tells me you had an always healing issue on a replicate vm volume, I'm having it and was hoping someone might know more than me about it?
03:31 dbruhn joined #gluster
03:34 smellis_ _dist: yep, hit that problem in my prod environment
03:34 smellis_ not in the lab
03:34 smellis_ this was through fuse mounts
03:34 smellis_ only tried gfapi in the lab environment
03:34 _dist smellis: I'm in test, just 1 vm. I'm running the vms over libgfapi, if they're on, they're healing. Same with fuse test though
03:34 _dist *vm
03:34 smellis_ I still have all of that lab stuff, should probly fire it up and see if I can repro in the lab
03:35 smellis_ hmmm
03:35 smellis_ I didn't see that and I ran up to 10 in the lab, which was all those lab machines could handle
03:35 _dist smellis: my main problem is, if I take a node down for maint, and they're all always healing, how can I ever know (I can't), when it's safe to take the other down
03:35 smellis_ I remember finding some traffic on the mailing list about this problem, I'll see if I can dig it up
03:36 smellis_ _dist: exactly why i'm running stand alone hosts right now
03:36 smellis_ lol
03:36 edong23 wait
03:36 smellis_ gluster is so close for this use case, but redhat doesn't officially support vms and bricks on the same servers
03:36 edong23 always healing?
03:36 edong23 oh i see
03:36 smellis_ so...
03:36 smellis_ lol
03:36 edong23 even after they finish "healing" from being down... they keep going into the heal state
03:37 smellis_ yeah, you never know when the cluster is in a consistent state across the board
03:37 _dist smellis: any idea, maybe you did more research, on how I can trace it, logs, etc? edong23 correct, they are always healing. You have to watch for split seconds when a vm image isn't healing on both that "means" that it's cool
03:37 smellis_ you it's funny, because I didn't see that in my lab, it would have been a show stopper
03:38 _dist smellis: was your lab combo hypervisors and storage?
03:38 smellis_ hell, even on GigE in another prod environment I run doesn't do this all the time
03:38 itisravi joined #gluster
03:38 smellis_ yep, bricks and vms on the same host
03:38 smellis_ plus an infiniband ring
03:38 smellis_ edong23: fuck you it's close enough to a ring
03:38 smellis_ lol
03:39 edong23 lol
03:39 _dist smellis: yeah that's where I'm at, my latency it's like 3-9ms maybe that's too much?
03:39 edong23 smellis_: we talked about this. It is a ring.. now that i understand you CAN go through a node to get to another
03:39 smellis_ edong23: ah ok
03:39 smellis_ that was the point of ospf
03:39 smellis_ lol
03:40 edong23 _dist: i dont  think there is a good way to test latency...
03:40 edong23 ping isnt good enough really...
03:41 _dist edong23: ah, well oddly enough (despite how I said I wouldn't) I ended up resorting to proxmox, it has a pveperf that does io latency to drives. Didn't research what it does for that though
03:41 edong23 _dist: but 3-9ms is outragous
03:41 edong23 lol
03:41 _dist ping is obviously < 1ms :)
03:41 edong23 oh ok
03:41 edong23 like   .09?
03:41 edong23 or .04 right?
03:42 _dist let me check now
03:42 edong23 _dist: im looking into proxmox hard
03:42 edong23 like its natasha nice porn
03:42 edong23 im not sure about it yet
03:42 edong23 but i got the demo and i want to check it out
03:42 _dist average 0.062ms over 15 pings
03:43 _dist edong: I ended up using it beause honestly I have issues that are too large with openstack, ovirt, archipel and virt-manager. I like libvirt (proxmox doesn't use it) but a pre-install iso on wheezy that comes with a compiled qemu for libgfapi and a gui for all of it, pretty nice I suppose
03:44 _dist at home I still run virt-manager
03:44 smellis_ _dist: what kind of volume do you have, rep or dist-rep?
03:44 _dist smellis: rep
03:44 smellis_ ok
03:44 _dist just 2 nodes right now
03:44 smellis_ so it's not something specific to dist-rep
03:44 edong23 we should all get a hotel room and bitch about all the options in open source virtualization
03:45 _dist :) whiskey would slow the progress
03:45 edong23 smellis_: you are dist-rep?
03:45 edong23 _dist: you assume tehre would ever be progress in such a discussion
03:45 smellis_ yeah, I wanted to be able to easily add another host, so 2 bricks per host
03:45 edong23 oh yeah
03:45 edong23 i remember
03:45 edong23 i have gotten scared of gluster
03:46 smellis_ I am confident it would work really well on your hardware
03:46 _dist smellis: I didn't have trouble converting a distribute to a replicate with a add-brick command, it actually synced in just a few min too
03:46 kanagaraj joined #gluster
03:46 edong23 smellis_: he is having issues and has more hardware than all of your office and mine combined
03:47 edong23 <_dist> edong23: I doubt that's the case in my situation. Mine is a test setup, single vm does it 24 cores, 300GB of ram on each box. I'm patient no rush just glad someone else has the same issue
03:47 smellis_ _dist: yeah either way I'd have to rebalance, which I am pretty sure would require a weekend maintenance window, just seemed to me to be less painfull if I start wil an even number of bricks
03:47 smellis_ yeah I read that
03:47 smellis_ but I know almost nothing about his environment
03:47 edong23 you think ill be ok on my quad core e3?
03:47 smellis_ _dist: how's your raw storage performance?
03:48 _dist smellis: yeah, I know what you mean. I'm in test so anything goes.
03:48 _dist smellis: it's alright, fsync writes between 500-700MB/s on each box
03:48 smellis_ what kind of vm?
03:49 smellis_ I know KVM, but is it winders or linux?
03:49 _dist in this case, it's a win7 (those writes aren't from within the guest). Thest gets 400/600 in crystal disk. I'm using latest virtio scsi
03:49 edong23 _dist: hardware raid or soft?
03:49 _dist edong23: soft, zfs
03:49 edong23 ew
03:50 edong23 no.. i kid
03:50 edong23 i have thought and thought about zfs
03:50 smellis_ what distro?
03:50 badone__ joined #gluster
03:50 _dist smellis: debian wheezy, but it's a redhat kernel (cause of proxmox)
03:50 edong23 but... i get faster speeds with raid 10... and i can get those speeds with raid 10 and have 3 copies of my data
03:50 smellis_ dammit, why do I have a _ after my name
03:50 smellis_ lol
03:51 smellis_ _dist: ah ok
03:51 edong23 smellis_: because freenode was dossed by foreign governments
03:51 _dist edong23: no ssds yet, just test so far smellis_: I'll take that into consideration
03:51 edong23 _dist: i dont have ssds either
03:51 edong23 20 1TB drives in md raid 10
03:51 smellis_ I haven't tried it on deb, have you tried centos?
03:51 edong23 smellis_: he uses proxmox
03:52 _dist edong23: you probably have writecache battery, or have turned off write caching
03:52 edong23 which, i have been looking at
03:52 smellis_ I take that back, i've tried the client on ubuntu against a centos base volume
03:52 _dist smellis_: I had the same issue on centos 6, and ubuntu 13.10
03:52 smellis_ gotchsa
03:52 edong23 _dist: no... no write cache... unless lsi hbas do it by default
03:52 smellis_ make's me want to revist the lab, still have all of dongers stuff at my office
03:53 edong23 i know, asshole
03:53 edong23 no, keep it and test
03:53 smellis_ yeah, I think I will
03:53 edong23 i have a few more weeks before ill need the pdu
03:53 smellis_ I still have those 10g cards too
03:53 edong23 but, i guess you dont really need it for the lab, right?
03:53 edong23 know
03:53 edong23 i was tring to sell them to _dist
03:54 smellis_ I don't really need the pdu, I doubt the cluster stuff is the problem
03:54 _dist unfortunately I want to go live soon, I might just switch back to debian with libvirt so I can at least do storage migration (ditch gluster until I figure this out)
03:54 smellis_ _dist: yeah, that's where I'm at
03:54 smellis_ copy-storage-all works on centos btw, altought not with a slick web interface like in proxmox :)
03:55 edong23 its weird that with a channel builing over with users... noone else has these issues
03:55 smellis_ edong23: I don't think this is a supported use case
03:55 _dist smellis_: how long did you troubleshoot? were you going down any paths I could follow?
03:55 edong23 VM storage?
03:55 smellis_ no, vms and bricks on the same hosts
03:56 edong23 eh...
03:56 edong23 see
03:56 edong23 why would it make a difference
03:56 edong23 in your case... lets say you are low on resources
03:56 edong23 but _dist is more def not...
03:56 _dist edong23: could be conflicts maybe, network, io locks, I don't know the ins and outs of qemu & gluster
03:57 nikk joined #gluster
03:57 edong23 network, i doubt...
03:57 edong23 not iwht 1 vm
03:57 _dist but I'm certain I saw the same thing with separate storage, but that was a long time ago
03:57 smellis_ _dist: I couldn't do much, because I had prod vms on it and couldn't do much at night, without hosing up the environment for the morning and had to capitalize on the time over the weekend to copy the vms out of the gluster volume
03:57 edong23 io locks... ok, but that wont be different with them on differen machines
03:58 _dist edong23: there was a bug for example where gluster ports overlapped libvirt migration ports for example. But that was 3.4.0, to say it was glusters fault I think it unfair as well
03:58 edong23 if smellis would give me his sata drives, i could dedicate 2 identical machines to this cause
03:58 edong23 lol
03:58 edong23 i remember
03:58 smellis_ Haha, I used a bunch of them in the two 2950s
03:58 smellis_ I only have two left
03:58 smellis_ lol
03:59 smellis_ for spares
03:59 edong23 i just dont trust... the sata drives i have...
03:59 edong23 not to be an intermediate storage server until testing is complete
03:59 edong23 and i cant buy 8-12 new drives...
03:59 edong23 i can... but its your tax money
03:59 edong23 hell, its mine too
04:00 smellis_ _dist: are you mounting the underlying fs with noatime and nodiratime?
04:00 smellis_ inode size of 512?
04:00 _dist smellis_: probably, it is the default. It's not just zfs, it's zfs --> zvol --> xfs --> gluster (so I can get directio for cache=none)
04:00 _dist yeah inode 512
04:01 _dist I suppose turning off noatime might help
04:01 smellis_ https://www.mail-archive.com/glust​er-users@gluster.org/msg13969.html
04:01 glusterbot Title: Re: [Gluster-users] One brick always has a lot of files that need to be heal (at www.mail-archive.com)
04:01 smellis_ read the reply from Pranith at the top, seems promising
04:01 smellis_ this might be why I don't see this on my other prod environment on 3.3
04:03 edong23 ah
04:03 nikk joined #gluster
04:03 edong23 _dist: might have to wait til 3.5
04:04 _dist yeah... I read that :) don't want to deal with it though
04:05 edong23 _dist: yeah... hm..
04:05 edong23 downgrade to 3.3?
04:05 _dist smellis_: looks like atime doesn't apply to zvols but good idea
04:06 smellis_ what about inode size?
04:06 _dist edong23: I could give that a try, means giving up libgfapi though, truthfully though I haven't seen massive speed increases from it
04:06 _dist smellis_: inode is xfs 512
04:06 smellis_ Oh, I thought it was on xfs
04:06 smellis_ zfs
04:06 _dist smellis_: but 4k at zfs level
04:06 smellis_ ah ok
04:07 edong23 yeah... smellis_ was expecting outrageous speed increases too, but.. we didnt see actual performance until we went 10gig and later infinerband
04:07 smellis_ I don't understand zfs then, lol
04:07 edong23 smellis_: think of it as the md part...
04:07 edong23 kinda
04:07 smellis_ gotcha
04:07 edong23 but with a logical volume manager
04:07 edong23 sorta
04:07 _dist smellis_: my block writes work out to about 20k if you do the calcs. yeah its complicated but it's like using md yeah
04:07 edong23 wait
04:07 edong23 wtf
04:07 edong23 _dist: you are using zfs in linux?
04:07 shylesh joined #gluster
04:07 _dist edong23: true
04:08 edong23 the non-open one?
04:08 edong23 or the open-one?
04:08 _dist edong23: wasn't aware there was a closed source one, running ZoL 0.6.2 I think is latest
04:08 edong23 i didnt mean closed... but it isnt gpl
04:09 _dist edong23: yeah, not the fuse one if that's what you mean. As far as I know, not GPL
04:09 sputnik13 joined #gluster
04:09 edong23 hm...
04:09 edong23 interesting though
04:09 edong23 im going to need your phone number
04:10 nikk joined #gluster
04:10 vpshastry joined #gluster
04:11 _dist edong23: it's really only a problem to ship solutions that are together from the start, a shame really
04:12 edong23 well, i dont kow the reasoning for the licensing differences.. but whatever.. i think zfs can be really good and the open source release will express the best parts
04:12 edong23 so, ill wait
04:12 edong23 ill do md for now
04:12 edong23 and noone scares me with "write-hole" problems...
04:13 smellis_ I just wish it was distributed too
04:13 smellis_ lol
04:13 _dist edong23: it's more likel brtfs will be ready before zfs is gpl
04:13 _dist at least in my opinion
04:13 edong23 there is already open-zfs though
04:13 edong23 its making good progress
04:14 edong23 btrfs is promising too
04:14 edong23 but it doesnt have all the features of zfs..
04:14 _dist not yet, but most planned
04:14 edong23 <smellis_> I just wish it was distributed too
04:14 edong23 you wish what was?
04:14 _dist smellis_: so you think 3.3 won't have this issue? I wouldn't mind running fuse if that fixed it
04:15 edong23 smellis_ has a production environment that is 3.3 i think   and it doesnt have this
04:15 edong23 and it is only on giggerbit
04:15 smellis_ _dist: not sure, but the other prod environment I'm running only has 3 vms on it and it's over GigE
04:16 edong23 i just said that
04:16 smellis_ oops
04:16 _dist smellis_: I don't want to abandon this design as a pipe dream, but I'm getting close to just going local storage, or perhaps trying another distributed storage with a new learning curve
04:17 edong23 _dist: i would say try 3.3 first
04:17 edong23 and highlight me with the results
04:19 smellis_ _dist: don't blame you, I went to local storage, the other dist storage systems require central metadata servers, so I didn't even try them, but I think ceph and sheepdog are good candidate if you have the time and will
04:19 sprachgenerator joined #gluster
04:19 smellis_ but I really think gluster is very close
04:19 _dist smellis_: yeap, just getting tired that's all. I know it's no the zfs component cause it does the same thing on native xfs for me
04:19 smellis_ and I've been running this same shared nothing arangment for over a year in two small prod environments
04:20 smellis_ _dist: understood, was just looking for something obvious that might be hurting performance
04:20 smellis_ not sure why I didn't see it in my lab though
04:21 _dist smellis_: your lab was 3.4.x?
04:21 smellis_ the lab also didn't have a bunch of busy winders server on it though
04:21 smellis_ yep
04:21 ngoswami joined #gluster
04:34 kdhananjay joined #gluster
04:39 rjoseph joined #gluster
04:40 ndarshan joined #gluster
04:44 ira joined #gluster
04:44 ppai joined #gluster
04:47 _dist smellis_: if you're still around, did you play with volume settings? see if they made a difference?
04:50 jporterfield joined #gluster
04:53 bala joined #gluster
05:01 rastar joined #gluster
05:01 jporterfield joined #gluster
05:06 meghanam joined #gluster
05:06 shyam joined #gluster
05:10 CheRi joined #gluster
05:12 saurabh joined #gluster
05:17 lalatenduM joined #gluster
05:17 haomaiwa_ joined #gluster
05:19 prasanth joined #gluster
05:21 eastz0r joined #gluster
05:24 davinder joined #gluster
05:29 hagarth joined #gluster
05:31 spandit joined #gluster
05:36 lalatenduM kkeithley, ping
05:40 lalatenduM glusterbot
05:40 rjoseph1 joined #gluster
05:41 nshaikh joined #gluster
05:44 dusmant joined #gluster
05:46 jporterfield joined #gluster
05:51 lalatenduM kkeithley, hagarth , I am getting error while updating from gluster3.4.2 to 3.5beta2, looks like repo issue http://fpaste.org/76034/92097019/
05:51 lalatenduM in Fedora 20
05:52 psharma joined #gluster
05:52 jporterfield joined #gluster
05:56 aravindavk joined #gluster
05:57 jporterfield joined #gluster
05:58 raghu joined #gluster
06:03 bulde joined #gluster
06:04 bulde :O
06:14 pk1 joined #gluster
06:24 _dist left #gluster
06:25 samppah :O
06:35 vimal joined #gluster
06:41 surabhi joined #gluster
06:49 PatNarciso joined #gluster
06:55 smellis joined #gluster
06:55 kshlm joined #gluster
07:14 pixelgremlins joined #gluster
07:19 jtux joined #gluster
07:20 ngoswami joined #gluster
07:20 mgebbe joined #gluster
07:25 ninkotech joined #gluster
07:25 ninkotech__ joined #gluster
07:26 PatNarciso joined #gluster
07:47 ctria joined #gluster
07:51 ktosiek joined #gluster
07:54 jporterfield joined #gluster
07:59 spiekey joined #gluster
08:06 micu joined #gluster
08:07 dneary joined #gluster
08:09 recidive joined #gluster
08:16 eseyman joined #gluster
08:18 blook joined #gluster
08:19 tryggvil joined #gluster
08:30 harish joined #gluster
08:34 hchiramm_ joined #gluster
08:49 andreask joined #gluster
08:53 calum_ joined #gluster
08:55 gdubreui joined #gluster
08:57 prasanth joined #gluster
09:00 keytab joined #gluster
09:11 liquidat joined #gluster
09:18 liquidat joined #gluster
09:30 lalatenduM hagarth, here is the 1st draft http://lalatendumohanty.wordpress.com/2014​/02/11/using-glusterfs-with-samba-and-samb​a-vfs-plugin-for-glusterfs-on-fedora-20/
09:32 ndevos lalatenduM: nice, but could you mention to put selinux in permissive mode, so that at least bugs can get reported and selinux will get improved over time?
09:33 ndevos lalatenduM: also, when mounting, it may be easier to mount as a guest user, no need to set passwords and all (just make sure to have write permissions on some dir on the volume)
09:42 blook joined #gluster
09:57 bulde joined #gluster
10:00 hagarth lalatenduM: looks good to me, johnmark one more blog to syndicate on gluster.org ^^^
10:18 bulde joined #gluster
10:22 davinder joined #gluster
10:34 lalatenduM ndevos, yup, agree will put some write up about it
10:35 JMWbot joined #gluster
10:35 JMWbot I am JMWbot, I try to help remind johnmark about his todo list.
10:35 JMWbot Use: JMWbot: @remind <msg> and I will remind johnmark when I see him.
10:35 JMWbot /msg JMWbot @remind <msg> and I will remind johnmark _privately_ when I see him.
10:35 JMWbot The @list command will list all queued reminders for johnmark.
10:35 JMWbot The @about command will tell you about JMWbot.
10:37 ekuric joined #gluster
10:38 spiekey joined #gluster
10:41 blook joined #gluster
10:48 prasanth joined #gluster
10:48 calum_ joined #gluster
10:50 mgebbe_ joined #gluster
10:51 92AAACY1Q joined #gluster
10:52 lalatenduM ndevos, ping
10:53 ndevos hey lalatenduM
10:54 lalatenduM ndevos, I have updated it with SELinux permissive info
10:54 lalatenduM ndevos, let me know if it is ok
10:54 ninkotech__ joined #gluster
10:54 diegows joined #gluster
10:54 lalatenduM ndevos, also need some information about the guest account to use it Samba
10:55 ppai joined #gluster
10:55 ndevos lalatenduM: 'man mount.cifs' should explain the guest account, I think it was 'mount -t cifs -o guest ...'
10:55 lalatenduM ndevos, checking
10:57 ndevos lalatenduM: have you re-published after the change? Its still listed as 'disabled' in the config file
11:00 lalatenduM ndevos, Yeah i haven't changed the config file but added text that we can either use  "SELINUX=disabled" or "SELINUX=permissive"
11:01 ndevos lalatenduM: ah, okay, I've read that, and was not sure how it was phrased before :) thanks!
11:01 davinder joined #gluster
11:01 lalatenduM ndevos, cool
11:03 tru_tru joined #gluster
11:05 franc joined #gluster
11:05 franc joined #gluster
11:08 CheRi joined #gluster
11:13 ira joined #gluster
11:15 ctria joined #gluster
11:18 klaas_ joined #gluster
11:19 kkeithley1 joined #gluster
11:20 prasanth joined #gluster
11:20 the-me joined #gluster
11:21 ninkotech__ joined #gluster
11:21 samppah joined #gluster
11:22 SteveCooling joined #gluster
11:27 jporterfield joined #gluster
11:57 ninkotech__ joined #gluster
11:58 meghanam joined #gluster
12:02 itisravi_ joined #gluster
12:07 failshell joined #gluster
12:13 itisravi joined #gluster
12:17 pk1 left #gluster
12:21 surabhi joined #gluster
12:22 hchiramm_ joined #gluster
12:54 failshell joined #gluster
13:03 pdrakeweb joined #gluster
13:18 dusmant joined #gluster
13:25 sprachgenerator joined #gluster
13:29 davinder joined #gluster
13:34 sroy joined #gluster
13:37 recidive joined #gluster
13:37 stickyboy joined #gluster
13:39 calum_ joined #gluster
13:42 haomaiwang joined #gluster
13:46 ninkotech__ joined #gluster
13:47 glusterbot New news from newglusterbugs: [Bug 1063832] add-brick command seriously breaks permissions on volume root <https://bugzilla.redhat.co​m/show_bug.cgi?id=1063832>
13:47 bfoster joined #gluster
13:48 plarsen joined #gluster
13:49 bet_ joined #gluster
13:50 haomaiw__ joined #gluster
13:50 pixelgremlins_ba joined #gluster
13:52 hagarth joined #gluster
13:52 pdrakeweb joined #gluster
13:55 B21956 joined #gluster
14:00 ppai joined #gluster
14:07 jmarley joined #gluster
14:08 rfortier joined #gluster
14:15 spiekey joined #gluster
14:15 lalatenduM kkeithley, ping
14:16 realsparticle joined #gluster
14:16 realsparticle anyone around to answer a few noob questions?
14:17 kkeithley_ lalatenduM: pong
14:17 realsparticle ping
14:18 kkeithley_ ,,(hello) | realsparticle
14:18 glusterbot kkeithley_: Error: No factoid matches that key.
14:18 kkeithley_ ,,(hello)
14:18 glusterbot kkeithley_: Error: No factoid matches that key.
14:18 kkeithley_ ,,(hi}
14:18 glusterbot I do not know about 'hi}', but I do know about these similar topics: 'hi'
14:18 kkeithley_ ,,(hi)
14:18 glusterbot Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
14:18 RameshN joined #gluster
14:19 nshaikh joined #gluster
14:20 lalatenduM kkeithley, http://fpaste.org/76034/92097019/, I am getting errors updating gluster rpms to 3.5beta2
14:21 kkeithley_ one sec while I look
14:21 realsparticle Ok here goes. I have a number of Ubuntu 12.04.3 NAS servers. They are of various sizes all with RAID arrays. Currently we replicate data across these NAS servers using rsync overnight. We are looking to provide multi site access to a common set of files and thought gluster may be an answer if we can install it on the existing servers and use the ex
14:21 realsparticle isting data/shares. Access would be via CIFS and native gluster client.
14:23 primechuck joined #gluster
14:23 realsparticle Files could be created amended at either site, so replication is very desirable for us.
14:23 RameshN_ joined #gluster
14:25 kkeithley_ lalatenduM: strange, the files are gone. Give me a couple minutes to re-upload them.
14:25 lalatenduM kkeithley, sure
14:25 realsparticle Also very new to irc so please be patient if I don't fully understand the protocol etc. :)
14:26 getup- joined #gluster
14:26 rossi_ joined #gluster
14:27 premera joined #gluster
14:27 lalatenduM realsparticle, the use case looks fine from GlusterFS point of view
14:27 lalatenduM realsparticle, you should try Gluster with it
14:28 Theory left #gluster
14:30 realsparticle lalatenduM
14:30 realsparticle lalatenduM
14:30 realsparticle lalatenduM - Just worried about any kind of data destruction. Can I install gluster configure a brick to point at one of our shares and then use it? Can I remove it without any destruction of our ext4 RAID arrays?
14:31 lalatenduM realsparticle, not sure if I understand your question, you want to use your exist NAS box shares as gluster bricks?
14:32 ihre joined #gluster
14:32 realsparticle I have a number of shares e.g. /media/NAS1/Documents/Staff Shared using CIFS are say NAS1Docs and mounted at the client using CIFS
14:34 realsparticle Ideally we would want to have 2 NAS servers one at each site configured using Gluster as replicated. They both have existing Ext4 RAID arrays with too much data to recreate!
14:35 realsparticle Ca we simply reuse the mounts that currently exist?
14:36 realsparticle So each Gluster Vol would be equal to our existing shares
14:39 realsparticle The shares on the existing NAS servers are shares using Samba. We have Linux and Windows clients
14:40 dbruhn joined #gluster
14:43 LessSeen_ joined #gluster
14:43 kkeithley_ lalatenduM: okay, files are uploaded
14:44 lalatenduM realsparticle, for that you have create gluster volume using clean bricks , mount the gluster volume and copy the contents of NAS1Docs to gluster volume
14:45 lalatenduM realsparticle, gluster can't recgnize existing files on the brick partitions
14:46 lalatenduM realsparticle, files should be created on gluster volumes either of the available client access methods
14:47 lalatenduM kkeithley, yeah I can see the files
14:47 lalatenduM kkeithley, thanks
14:47 ninkotech_ joined #gluster
14:50 realsparticle lalatenduM Really confused now..... I thought Gluster sat above the 'real' filesystem in this case Ext4
14:56 rwheeler joined #gluster
14:57 kkeithley_ realsparticle: there are some tricks people have for creating gluster volumes using "bricks" that are already filled with files. But officially we don't support that model of deployment.
15:00 kkeithley_ and yes, gluster does sit above the real file systems. It's just like NFS in that respect.
15:02 realsparticle kkeithley OK thanks. Still confused. Does that mean that If I create a new set of 'bricks' on the NAS servers I would need both a mount point using Gluster and a mount point using CIFS to copy/polulate the 'bricks' Once that is done I could delete the original mount points and the data contained in them. Then I would need to point the samba shares
15:02 realsparticle at the new Gluster 'bricks' on the servers so that Samba was now sharing the new 'bricks'.
15:04 kkeithley_ just like NFS and Samba in that respect. And yes, that's the gist of what we officially support  You create an empty gluster volume and fill it up from a "client" mount point.
15:05 jbrooks joined #gluster
15:08 ninkotech_ joined #gluster
15:09 realsparticle kkeithley OK, maybe have to have a rethink. The existing data is large. Not sure we have the capacity to do a parallel setup. Can you provide info on the tricks you mentioned so we could do some testing. Is removing Gluster destructive to the data on the NAS Ext4 arrays?
15:10 kkeithley_ removing gluster never destroys your data.
15:11 kkeithley_ Maybe JoeJulian, when he comes on-line a little later, can tell you more about creating bricks that are prefilled with data. Or check out his blog at joejulian.name.
15:12 kkeithley_ @later tell JoeJulian tell realsparticle about tricks for creating volumes full of data
15:12 glusterbot kkeithley_: The operation succeeded.
15:15 realsparticle kkeithley OK I will have a look. It sounds like we could try Gluster and then if the performance didn't work for us we could remove Gluster and remount the 'bricks' with CIFS and Samba
15:15 realsparticle Losing no data
15:16 dbruhn The unsupported method of pre filling a brick is to have your left most brick have the data on it, that is the only one that will work.
15:16 bugs_ joined #gluster
15:19 dusmant joined #gluster
15:21 realsparticle dbruhn And then the right brick will replicate?
15:22 wushudoin joined #gluster
15:22 dbruhn Then the self heal ends up populating the replicant, from my understanding.
15:23 natgeorg joined #gluster
15:23 natgeorg joined #gluster
15:24 realsparticle dbruhn Not really an answer fo rus then it seems as trying to replicate over 30TB of data on our WAN link would be very very bad :(
15:24 dbruhn Ahh, sorry I missed a good chunk of the conversation I think.
15:25 dbruhn Why would you want to use synchronous replication over a wan link in the first place?
15:26 realsparticle We want to have access to files from both sites in real time. Both sites are also creators/editors
15:26 vpshastry joined #gluster
15:26 dbruhn Would you be using the gluster client or NFS
15:27 kkeithley_ He's using Samba/CIFS
15:27 realsparticle A combo of the Gluster client and CIFS as we have windows clients also
15:27 dbruhn Ahh ok.
15:27 internet_analyze joined #gluster
15:27 internet_analyze hello
15:27 glusterbot internet_analyze: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an
15:27 glusterbot answer.
15:27 realsparticle No real other options I don;t think
15:28 dbruhn The reason I ask is because the Gluster client when writing to the replicant pairs it writes to both at the same time.
15:28 dbruhn Reading is based on first response by default
15:29 realsparticle Which would be the local NAS unless down
15:29 dbruhn Understood, just pointing out your writes will be just as time consuming
15:29 ale84ms I got an issue with my gluster servers... We changed the IP addressing of our servers (included the machines where the bricks are located). How can I change the IP address of the bricks?
15:30 realsparticle Any idea the % overhead
15:30 ihre Would it be possible to create an overlay from 2 bricks, one as read only and the other with read write? (I'm looking for a way to replace unionfs-fuse, http://dpaste.com/1605117/ )
15:30 glusterbot Title: dpaste: #1605117 (at dpaste.com)
15:32 realsparticle Sorry, are you saying write will not complete until both sides are wirtten?
15:36 failshell joined #gluster
15:40 dbruhn realsparticle, sorry just am in phone
15:40 realsparticle Hey no worries, kust grateful for the support/insights
15:42 dbruhn realsparticle, you are correct the write won't complete until it's written to both machines in a replicant pair from the gluster client. I am not sure how it works with the NFS client, the NFS client is really just another gluster client, and the symmetrical writes happen after it's introduced to the NFS server.
15:43 realsparticle We don't use NFS at all.
15:43 dbruhn ale84ms, you will need to do a peer probe on one of the servers to the one you changed the ip of
15:44 dbruhn So CIFS is doing essentially the same thing, as it is just a front to the gluster client
15:45 dbruhn ihre, not sure what you are asking. But it sounds like you want to have a real time replication that doesn't allow the users to interact with it other than read it?
15:46 realsparticle Ah ok. Any idea the % overhead for Gluster. It sounds like the WAN latency is the real bottleneck. We have a 100MB connection between sites.
15:46 realsparticle Usually <10ms ping
15:46 dbruhn It could become a problem for sure.
15:46 ale84ms I did "gluster peer probe 192.168.5.6" from one of the two machines in the gluster, but I got "peer probe: success: host 192.168.5.6 port 24007 already in peer list"
15:46 dbruhn A lot of guys use geo replication for site to site stuff
15:47 ale84ms and if I run "gluster peer status", it keeps the previous IP and "State: Peer in Cluster (Disconnected)"
15:47 dbruhn ale84ms, that's good, now probe back from the other server
15:47 realsparticle ??? would need to do some more reading I think. But which ever way we still need to replicate files in 'real' time between the sites
15:48 dbruhn ale84ms, you might need to restart the services after
15:48 ale84ms note that I changed the IP on both machines, from 192.168.178.0/30 to 192.168.5.4/30. The two machines communicates with a back-to-back ethernet connection
15:48 ale84ms ok, I will try now
15:49 dbruhn realsparticle, I am not saying it won't work, there are guys using replication between sites the same way you are proposing, just wanted to make sure you knew what was involved.
15:49 realsparticle Just trying to get my head around the possibilites
15:50 dbruhn ale84ms, you will need to re-probe bother servers then.
15:50 dbruhn realsparticle, absolutely
15:50 realsparticle Initially when I started reading about Gluster my reaction was Eureka! :)
15:51 ale84ms I did "/etc/init.d/glusterfs-server restart" on both machines after probing each other, but I still got the previous IPs
15:51 primechuck realsparticle:  It will work in theory, if you don't mind writes taking as long as the round trip is between your two sites. :)
15:52 ale84ms I forgot to tell... I'm running on both servers Debian Lenny
15:52 dbruhn You might just need to justify a bigger lower latency pipe
15:52 dbruhn ale84ms, glusterd is the process you want to restart
15:53 realsparticle Not possible I'm afraid :( Sounds like the larger the file the worse things get
15:53 realsparticle We have quite a lot of media files being created edited
15:54 olisch joined #gluster
15:54 dbruhn edited at both locations? or is one side more of a read/review
15:54 realsparticle No both. These 2 sites are Colleges
15:54 ale84ms shouldn't the restart of glusterfs-server trigger the restart of glusterd? It says "[ ok ] Stopping glusterd service: glusterd.
15:55 ale84ms [ ok ] Starting glusterd service: glusterd." right after the command is triggered
15:55 dbruhn ale84ms, nope
15:55 dbruhn realsparticle, if that's the case the gluster client is going to solve another problem you have, because it's going to ensure the files are the same on both sides.
15:56 ale84ms how can I restart glusterd service?
15:56 dbruhn Anything that handles the replication after the fact could become a race condition
15:56 realsparticle Indeed. We are facing a growing nomadic population between the sites
15:56 dbruhn /etc/init.d/glusterd restart
15:57 ale84ms I don't have glusterd in /etc/init.d
15:57 dbruhn ale84ms, what version of gluster are you running?
15:58 ihre dbruhn: you've translated it the way I couldnt, but yes, that is basically what I am looking for. At the moment we combine different 2 directories(branches, 1 ro, other rw) as 1 mountpoint where writes are merged/copied to the rw branch.
15:58 ale84ms glusterfs 3.4.0 built on Jul 19 2013 03:56:40
15:59 vpshastry joined #gluster
16:01 dbruhn ihre, I am not sure gluster is the right tool for what you want to do. It will handle the synchronous replication of the data, but anything introduced to the volume is going to end up on both replicant pairs.
16:02 dbruhn ale84ms, what is the output of "ps -ef | grep gluster" use fpaste
16:02 ninkotech__ joined #gluster
16:03 ale84ms fpaste?
16:03 dbruhn http://fpaste.org
16:03 glusterbot Title: New paste Fedora Project Pastebin (at fpaste.org)
16:03 dbruhn so you don't spam the channel and get banned
16:03 ale84ms ok, let me check it
16:04 ale84ms ps -ef | grep gluster
16:04 ale84ms root      7512     1  0 11:19 ?        00:00:02 /usr/sbin/glusterfsd -s 192.168.178.2 --volfile-id testvol.192.168.178.2.storage-brick -p /var/lib/glusterd/vols/testvol/ru​n/192.168.178.2-storage-brick.pid -S /var/run/593ca3342bfa29b8f74cb2821ffde18f.socket --brick-name /storage/brick -l /var/log/glusterfs/bricks/storage-brick.log --xlator-option *-posix.glusterd-uuid=cd3b71f​8-ded1-4423-a9fc-65b2522de54a --brick-port 49152 --xlator-option testvol-se
16:04 ale84ms root     11197     1  0 16:51 ?        00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
16:04 ale84ms root     11250  9978  0 17:02 pts/6    00:00:00 grep gluster
16:04 ihre dbruhn: thanks for the short explanation, I'll take a look at cow filesystems.
16:05 vpshastry left #gluster
16:05 ale84ms sorry... http://ur1.ca/glyl2
16:05 glusterbot Title: #76162 Fedora Project Pastebin (at ur1.ca)
16:05 dbruhn ale84ms, I guess you could reboot your servers. glusterd is running from the looks of it, not sure what you are using for a script to start it
16:06 ale84ms ok, I will reboot them now. Give them some minutes to wake up once again, and I'll tell you the situation
16:08 smellis joined #gluster
16:10 davinder joined #gluster
16:12 rfortier1 joined #gluster
16:12 recidive joined #gluster
16:14 ale84ms ok, now that I try to start the gluster with "/etc/init.d/glusterfs-server start" I get "[FAIL] Starting glusterd service: glusterd failed!"
16:15 ale84ms on both machines... it didn't start automatically after rebooting the system
16:15 shyam joined #gluster
16:15 realsparticle dbruhn, many thanks for your help. I need to have another look through the docs and then come back with more specific questions. Cheers
16:15 realsparticle Spart
16:16 failshell this volume has been georeplicating for 2 weeks now. over 15 millions files so far ..
16:17 dbruhn ale84ms, can you check the log files for gluster "/var/log/glusterfs/"
16:17 calum_ joined #gluster
16:17 dbruhn there are several in there and see why it's not starting
16:17 wralej joined #gluster
16:18 wralej Does gluster have the concept of a replication network, separate from the client network?  (I'm planning my VLANs)
16:18 dbruhn wralej, are you using the gluster client or ifs?
16:18 dbruhn nfs
16:18 failshell wralej: no
16:19 wralej dbruhn: both, i think.  i'll be using ovirt
16:19 failshell no if you use gluster clients
16:19 wralej (ovirt hosted engine, which at this time seems to require nfs)(
16:19 failshell wralej: some people monkey around with a seperate vlan, but it doesn't change much
16:19 dbruhn wralej, the gluster client actually handles the replication, some people do some magic using dns to split traffic out in other ways, but for what you are talking about you would only be able to do that through NFS
16:20 failshell in my case, i just dismantled that seperate VLAN because it didn't improve anything in our case
16:20 dbruhn +1 failshell
16:20 dbruhn if you need lower latency or improved throughput, a network upgrade is usually a better option
16:20 failshell wralej: what you want is a fast network.
16:20 failshell i run gluster on a 1Gbps network and well, its not fast
16:21 failshell i can write a 50MB/s using gluster clients
16:21 failshell wish my employer would budget for a 10Gbps network
16:21 wralej ah.. that's cool. okay.. i think i'll skip the vlan for that then.. thanks dbruhn and failshell.  i'm going to be running 10GbE, but I'll have virtualization networks on that port too
16:21 failshell 10GbE should give you something around 500MB/sec for writes. similar to a SSD
16:22 ale84ms I tried to restart the services while reading "/var/log/gluster/etc-glusterfs-glusterd.vol.log" and I got this http://fpaste.org/76172/13921356/
16:22 glusterbot Title: #76172 Fedora Project Pastebin (at fpaste.org)
16:22 _dist joined #gluster
16:22 wralej i wish i had two 10GbE ports, but budget fell over.. cool.  i think i'll be disk bound at that point.. i have 2 * 3TB Seagate HDD and 1* 240GB Samsung SSD, but the SSD will be the OS and such...
16:22 dbruhn ale84ms, is the partition containing /var/lib/glusterd full
16:23 wralej (per each of 5 nodes)
16:23 dbruhn I am running QDR Infiniband for latency reasons. I wish I had FDR!
16:23 dbruhn lol
16:23 wralej nice..lol.. i miss infiniband
16:23 failshell wralej: i recall reading somewhere you need an even number of nodes
16:24 failshell but dont quote me on that
16:24 wralej failshell: that's not good.. i'll have to look in on that.. thanks for the heads up
16:24 ale84ms no, there are more than 80GB free
16:24 sprachgenerator joined #gluster
16:24 dbruhn ale84ms, can you output "df -h" into fpaste.org
16:25 failshell wralej: i hope for you  you don't have millions of files to import
16:25 wralej failshell: i don't.. blank slate
16:25 sprachgenerator joined #gluster
16:25 ale84ms http://ur1.ca/glypd
16:25 glusterbot Title: #76175 Fedora Project Pastebin (at ur1.ca)
16:26 dbruhn ale84ms, something is wrong with your volume files. since we just restarted the servers we just flushed a bunch of logs and the /tmp directory. I've seen gluster corrupt the log files if the volume they are stored on becomes full.
16:27 failshell anyone using CTDB here?
16:28 failshell for NFS/SMB volumes?
16:28 ale84ms I'm pretty sure that none of the volumes on any of the two machines became full
16:29 ale84ms is there any way to restore the volume files?
16:30 dbruhn Did you have them backed up?
16:30 wralej failshell: seems an odd number of servers makes for difficulty in reasoning about 2-replica "chain"...
16:30 wralej whatever that means
16:31 failshell ive always built clusters with even numbers of nodes
16:31 failshell all our volumes have 2 replicas
16:31 dbruhn wralej, if you have to manage servers in a replication symmetry becomes important to the logical config.
16:31 _dist I was wondering if anyone here is using gluster for VM storage on 3.4.x and isn't having the "always healing" issue.
16:31 ale84ms no... I don't have any backup
16:32 wralej dbruhn, failshell: sounds good.. thanks for the help
16:32 dbruhn ale84ms, how much data do you have on the volume?
16:32 dbruhn and did you build the system?
16:33 theron joined #gluster
16:34 primechuck _dist:  With 3.4.x the healing algo is always doing a check on the large VM files, but never locks them for healing.  When it does the check it shows as healing in the output for some reason.  Though the additional diff instead of full heal is a must have ;)
16:34 primechuck It was freaking me out for a while when I thought it was actually doing a heal on the files
16:35 ale84ms about 4.3 TB, and most of them are small files
16:35 _dist primechuck: if that's the case, I'd be pretty happy. However, if it is then I'm back to the question "how do you ever know when a file is actually healing, vs _fake_ healing?"
16:35 ale84ms I built the system personally... but I didn't have much knowledge about gluster. I followed the instructions on the website though...
16:37 _dist primechuck: I've got a setup I'm very happy with, but I can't imagine going live with 30 vms, taking a node down for maint and never knowing when things are truly _healed_ after it's up. How do you deal with that?
16:42 ninkotech__ joined #gluster
16:45 _dist primechuck: no? :)
16:46 primechuck If the healing algo actually repairs something, volume heal VOLENAME info healed or heal-failed will log it.  If the Algo is in the process of checking, it appears in appears in the output of gluster volume heal VOLNAME info
16:46 primechuck We've just had to live with the fact that info really means, file being checked, and parse the logs accordingly
16:47 _dist primechuck: we likely have something else then. 100% of my machines will show up in volume heal info in a given 10 second span. Oh ok, so you do have that?
16:48 _dist primechuck: So you use volume heal info healed (as your gold standard, assuming any machine that was running when the node went down, needs to show up there before you are safe)
16:48 primechuck IIRC the self heal aglo does a quick check on the write to a file, which for a VM image is a lot.
16:49 primechuck I'll have to find the RegEx, but we parse the client and the brick logs to look for failures.
16:49 dbruhn ale84ms, what kind of volume?
16:50 _dist primechuck: your script, or one out there? The only issue if I have with the volume heal info healed is that you have to know which machines to be looking for, some may have never touched their HD in the downtown so you'd be waiting forever
16:50 semiosis each file has ,,(extended attributes) which indicate if the file needs healing.  a self heal check looks at those xattrs, and only when the xattrs indicate a heal is needed does glusterfs actually look at the file data
16:50 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
16:53 _dist semiosis: thanks for piping in, is what I'm seeing normal? I'm getting mixed responses and I can't find anyone using gluster for VM storage that doesn't have it (I want to find someone obviously)
16:53 ale84ms what do you mean? I cannot check the volume status now 'cause the gluster daemon is not operational
16:53 kaptk2 joined #gluster
16:53 ale84ms I had two brick, if I remember well I replicated data on bricks
16:54 primechuck when the AFR does a an xattr check, it shows it as healing in the output of gluster volume heal info
16:54 primechuck Is that not supposed to be the case?
16:55 dbruhn ale84ms, gluster doesn't start on either server?
16:55 semiosis _dist: idk whats normal for VM hosting.  JoeJulian might
16:55 ale84ms exactly
16:55 semiosis _dist: he hosts vm images on gluster, i dont
16:55 sprachgenerator joined #gluster
16:56 _dist semiosis: Yeah when I spoke to JoeJulian he had the same "issue" that was a while ago
16:57 semiosis _dist: the real test is to set it up & cause a failure... does it work?
16:57 semiosis i.e. does the VM fail over without data loss/corruption on the image
16:57 semiosis if you can repeat that successfully, then i wouldnt worry about what the command output says
16:57 ale84ms on both machines with "/etc/init.d/glusterfs-server start" I get "[FAIL] Starting glusterd service: glusterd failed!"
16:58 _dist semiosis: nope, it doesn't appear to fail, even if I'm doing something like a file copy in the guest
16:58 semiosis well then
16:58 _dist semiosis: I am worried though! beause there is no definitive way to tell if it's a "fake" heal, or a real one. The volume heal info healed helps, but not 100%
16:58 semiosis _dist: write up what you did & put it on the wiki as a guide to others!  if you'd be so kind :)
16:59 _dist semiosis: Sure I could do that, I just wanted to make sure that no-one out there with experience didn't contradict mine, I'm relative new (4-5 months) to using glusterfs for VMs
16:59 semiosis _dist: as long as both servers are online there is no need to heal, because the client writes to all replicas in sync at the same time.  only when one replica is offline will a "real" heal be needed
16:59 semiosis hope that allays your fear
17:00 semiosis _dist: put it on the wiki and they'll let you know if they disagree :)
17:00 _dist semiosis: Agreed, but let's say I've got 2 nodes. I need to dist upgrades on both, I take down node a, do the upgrade and power him on. There is a finite amount of time I have to wait before taking down B, I'd be afraid to take down B today
17:01 Matthaeus joined #gluster
17:01 semiosis _dist: you can stat the file through a fuse client and the fuse client logs will tell you if a heal is needed and when it has completed
17:01 semiosis _dist: if you stat the file and the fuse client log doesnt mention heal needed, the file is good
17:01 semiosis _dist: if it mentions a heal is needed it should also say when the heal is completed
17:01 semiosis afaik
17:01 semiosis you should double check what i'm saying though
17:02 _dist semiosis: yeap, always would, no offense intended. I'm usually very careful :)
17:03 semiosis _dist: you can also check the ,,(extended attributes) yourself on the backend brick filesystems -- if the AFR xattrs are all-zero then the file does not need healing (is totally consistent)
17:03 glusterbot _dist: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
17:03 daMaestro joined #gluster
17:03 quique_ joined #gluster
17:04 daMaestro|isBack joined #gluster
17:06 PatNarciso joined #gluster
17:07 JoeJulian _dist, semiosis: on any file that's accessed a lot, you're going to see dirty xattrs and have it show up in heal occasionally (or frequently depending on how busy). That just means it's in a pending state, not necessarily that it actually needs healed.
17:07 _dist JoeJulian: Thanks, so, how do you keep track yourself?
17:07 primechuck One our 6 server gluster, rolling updates are done one not at a time, then a stat all the files on the volume via a fuse client, then move on to the next server.  IIRC the logs from glustershd.log show the status.
17:08 primechuck JeoJulian: The output of heal info is files pending a check?
17:08 JoeJulian _dist: The process is, increment pending data xattr, write data, decrement pending data xattr. If you happen to poll right in the middle of that, it'll show up in the list.
17:08 primechuck Well xattr check?
17:09 _dist JoeJulian: Ah, so that's why a watch -n1 pretty much always shows it. So broken record and all that, how do I know :)
17:09 JoeJulian In nagios I just set a high sample threshold. I've also got cattle instead of kittens.
17:09 vpshastry joined #gluster
17:10 JoeJulian primechuck: I haven't actually read the code on that, but from my crude analysis that seems to be so.
17:10 primechuck Doesn't the AFR wrong to glustershd.log if it actually detects an error and corrects the data?
17:11 JoeJulian Yes, but a client may log that as well.
17:11 _dist JoeJulian: if that answer was for me, I'm not following :)
17:12 nage joined #gluster
17:12 nage joined #gluster
17:12 JoeJulian file a bug report. That report is broken. Maybe there should be a check against the active iops to ensure that it's really in need of healing.
17:12 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
17:14 JoeJulian Frankly, when I'm doing a brick replacement and I want to make sure the replication is complete, I still do my old python script that checks xattrs on the source bricks.
17:14 JoeJulian I walk the tree once, collecting dirty xattrs. I walk the dirty list a second time to see if they've cleared.
17:15 zaitcev joined #gluster
17:16 primechuck JoeJulian: Do you do one brick per HD or do you have raid underneath it for you VMs?
17:17 JoeJulian I do lvm partitions breaking up each HD. I do multiple volumes and have one brick per hd for each volume.
17:17 JoeJulian My configuration is for fault tolerance, not performance. My needs are very small.
17:18 xavih _dist: There is a bug report for that problem: https://bugzilla.redhat.com/show_bug.cgi?id=912239
17:18 glusterbot Bug 912239: unspecified, unspecified, ---, vsomyaju, POST , "gluster volume heal volname info" shows files which are under modification and may not require self-heal.
17:19 xavih _dist: however there is no solution yet
17:19 _dist xavih: thanks
17:19 JoeJulian Thanks xavih
17:20 xavih yw :)
17:20 JoeJulian Hmm, claims to be in post.
17:21 daMaestro joined #gluster
17:21 _dist JoeJulian: So right now you check the xattr as semiosis suggested, but look for the specific pattern rapidly?
17:21 xavih JoeJulian: there is a patch in review, but it seems that it's not quite acceptable: http://review.gluster.org/#/c/4534/
17:21 glusterbot Title: Gerrit Code Review (at review.gluster.org)
17:22 xavih JoeJulian: I don't know if there is any other patch to solve it
17:22 dbruhn ale84ms must have left
17:22 jayo joined #gluster
17:22 JoeJulian That was a year ago... <sigh>
17:23 bstr left #gluster
17:23 xavih yes...
17:23 JoeJulian There are no other entries for that bug either.
17:23 matclayton joined #gluster
17:23 matclayton Is it safe to rsync a worm only brick to recover a lost brick
17:24 JoeJulian As long as you don't cross the streams. That would be bad.
17:25 JoeJulian matclayton: is this brick replicated?
17:25 matclayton yes
17:25 primechuck JoeJulian:  Cool.  How does your configuration handle things like, a drive occasionally doing into a blocked IO state when it is failing?
17:25 KyleG joined #gluster
17:25 KyleG joined #gluster
17:25 matclayton JoeJulian: We lost a 45TB raid6 array, and every time we bring it back online to let the shd recover it, it takes down the entire site
17:26 matclayton we have nginx serving files from the volume, and it locks up badly
17:26 KyleG left #gluster
17:26 JoeJulian primechuck: Was horrible. Haven't had that happen since the workaround was added to 3.4 so I'm not sure yet.
17:26 KyleG joined #gluster
17:26 JoeJulian matclayton: Ah, right. The too-many self-heals problem...
17:27 matclayton JoeJulian: sounds about right
17:27 matclayton JoeJulian: I suspect we've run out of background heals
17:27 primechuck JoeJulian:  Glad I'm not the only one :)  Are you talking about the brick failure detection in 3.5?
17:27 matclayton any more suggestions. Right now we've killed the brick process and its fine, until we bring it back online. This is gluster 3.3.1
17:28 JoeJulian primechuck: Did it not make 3.4... I wasn't following that too closely.
17:28 primechuck JoeJulian:  It was on the 3.5 feature list, I don't know if it made it to the backport list.
17:29 semiosis backport wishlist, feel free to add your wishes: http://www.gluster.org/community/docu​mentation/index.php/Backport_Wishlist
17:29 glusterbot Title: Backport Wishlist - GlusterDocumentation (at www.gluster.org)
17:29 JoeJulian matclayton: My idea would be to use iptables on the new brick server to block the clients from reaching the brick. Let the self-heal daemon handle it, or mount the volume on a non-critical machine that you allow access and to a find.
17:29 JoeJulian s/to a/do a/
17:29 glusterbot What JoeJulian meant to say was: matclayton: My idea would be to use iptables on the new brick server to block the clients from reaching the brick. Let the self-heal daemon handle it, or mount the volume on a non-critical machine that you allow access and do a find.
17:30 primechuck JoeJulian:  I just worked around the problem with raid and aggressive offlining of bricks that have high blocking IO.
17:30 matclayton So the clients and servers run on the same boxes
17:30 JoeJulian matclayton: That would prevent the critical clients from getting blocked by foreground self-heals, but would still allow gluster to do the repair.
17:30 JoeJulian matclayton: You like to cause trouble don't you. ;)
17:30 matclayton JoeJulian: sadly yes
17:32 Mo_ joined #gluster
17:33 theron joined #gluster
17:33 JoeJulian matclayton: Well.... since the new brick/nginx server are currently offline... You could still use iptables to block the other machines. Then either let the self-heal daemon on the new brick server perform the heal (since it'll still be able to access all the bricks) or mount the volume locally on that machine and do a find.
17:33 JoeJulian matclayton: Once you're all synced up, you can open up the ports again.
17:34 JoeJulian That's how I would do (and have done) it.
17:34 matclayton JoeJulian: Sounds like a plan... just trying to figure out the details here
17:34 JoeJulian @ports
17:35 glusterbot JoeJulian: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
17:35 semiosis JoeJulian: could matclayton just rsync the data over then bring it online?
17:36 JoeJulian That should help the data heal, but I'm not sure what the xattrs would do if you copy the pending flags.
17:36 matclayton JoeJulian: pending flags?
17:36 JoeJulian @extended attributes
17:36 glusterbot JoeJulian: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
17:36 matclayton JoeJulian: we were thinking syncing including the xattrs
17:37 KyleG JoeJulian: Do you generally lose a lot of data using #gluster? I idle here pretty much to look for situations like yours, in consideration of one day using gluster.
17:37 semiosis JoeJulian: ooh right, good call
17:37 dbruhn joined #gluster
17:38 cp0k joined #gluster
17:38 JoeJulian Right, but your good brick has xattrs that show there are pending writes destined for your server that's down. If you copy those xattrs would it think it's split brain since it has xattrs claiming it has pending writes for itself? I'm not sure.
17:38 JoeJulian It
17:38 semiosis this would be interesting to test
17:38 JoeJulian It's something worth trying in a test environment, but with your production stuff I'd rather be careful.
17:39 semiosis the xattrs do indicate which brick has the dirty files, so it might work ok
17:39 cp0k hey gangs, I am happy to report my upgrade to Gluster 3.4.2 was a complete success yesterday! :) I was shitting bricks during the process, no pun intended haha
17:39 cp0k s/gangs/gang/g
17:39 glusterbot cp0k: Error: u's/gangs/gang/g hey gangs, I am happy to report my upgrade to Gluster 3.4.2 was a complete success yesterday! :) I was shitting bricks during the process, no pun intended haha' is not a valid regular expression.
17:39 JoeJulian hehe
17:39 cp0k s/gangs/gang/g
17:40 cp0k 127.0.0.1:/storage  189T  161T   19T  90% /storage
17:40 cp0k is the reason why
17:40 cp0k would have been very very very bad to loose all that data somehow
17:41 cp0k anyways, one thing I still notice is that at times CPU0 on my storage nodes spikes to 100% for a while at a timne
17:41 cp0k the spike is in Wait-IO
17:41 cp0k http://cl.ly/image/3N2d1I2P3p0q
17:41 glusterbot Title: Screen shot 2014-02-11 at 12.41.31 PM.png (at cl.ly)
17:42 cp0k rest of the CPUs are fine, the behavior was the same prior to the upgrade
17:43 JoeJulian Get faster disks?
17:45 tyrfinged joined #gluster
17:46 cp0k JoeJulian: this is for CDN origin, performance is not an issue.
17:47 cp0k JoeJulian: I was just curious if there is anything I may be able to tune, if not no biggie, as it is not actually causing any issues
17:47 JoeJulian Nope. Waiting for drives is waiting for drives.
17:47 cp0k Understandable of course, but this is only happening on CPU0
17:47 JoeJulian The only way to cure that is to not wait for drives.
17:48 KyleG I was going to say what about your L2Arc
17:48 cp0k rest of the CPUs don't have this issue
17:48 KyleG but then I was like uhps I'm in #gluster
17:48 KyleG lol
17:48 JoeJulian lol
17:48 jobewan joined #gluster
17:48 JoeJulian If you haven't, you might try deadline.
17:50 internet_analyze joined #gluster
17:50 internet_analyze dbruhn, I managed to solve my problem
17:51 JoeJulian KyleG: are you perhaps running xen?
17:51 ale84ms dbruhn, sorry... I got the wrong username. I managed to solve my problem
17:51 KyleG JoeJulian: Nah. I'm ESXi.
17:51 ale84ms basically, I needed to replace every previous IP with the new IP in every single file inside "/var/lib/glusterd"
17:52 jayo left #gluster
17:52 ale84ms including every single file in every single directory
17:52 ale84ms then everything works fine, I can restart the server and everything connects and works
17:52 cp0k ale84ms: What was your original problem? I was not here for it :/
17:53 matclayton JoeJulian: We've setup the firewall and touching files now, seems to be working
17:54 twx joined #gluster
17:54 ale84ms cp0k, sorry! dbruhn followed me before... Basically, I had to change the IP addresses of every server in the gluster
17:54 wushudoin left #gluster
17:54 cp0k ah okay
17:55 ale84ms and I didn't know how to make the gluster to see the new network configuration and connect/work once again
17:55 rotbeard joined #gluster
17:55 JoeJulian @hostnames
17:55 glusterbot JoeJulian: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
17:55 ale84ms he suggested me to probe the new IPs, but it didn't work...
17:55 ale84ms neither a reboot worked, or the restart of the gluster daemon
17:57 JoeJulian That's why we use hostnames. :/ To change an ip address of a host if you didn't use hostnames, the only supported way is to remove the volume, disassociate the peers, change your ip addresses, probe the peers, re-create your volumes.
17:58 JoeJulian The unsupported way is to stop your volumes, stop glusterd, replace the ip addresses in every file they occur under /var/lib/glusterd on all your servers, start glusterd, start volumes.
17:59 purpleidea btw, fwiw you can ever use "fake" domains such as "*.example.com" for the hosts in your gluster pool as long as it's all internal. i use this by default for my gluster testing pools.
17:59 purpleidea s/ever/even/
17:59 glusterbot purpleidea: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
17:59 JoeJulian You can use no domain and only shortnames even.
18:00 JoeJulian s/example.com/example.local
18:00 purpleidea oh! well or that even! ^^^ although without a fqdn, i did have a problem with something. maybe it was puppet. i forget.
18:00 cp0k JoeJulian: you mention that the only supported way  is to remove the volume, disassociate the peers, etc... then recreating the volume.
18:00 cp0k JoeJulian: Wouldnt this lead to data loss?
18:00 cp0k or not so much since Gluster has a hash to every file?
18:01 JoeJulian cp0k: nope, though you will trip over the "path or prefix" is already part of an existing volume thing...
18:01 JoeJulian @path or prefix
18:01 glusterbot JoeJulian: http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
18:01 cp0k I have never tried deleting a volume then recreating it with the same peers / bricks in the same order so I dunno what the result of that would be
18:02 cp0k interesting :)
18:02 cp0k I like this glusterbot fella
18:02 JoeJulian brb
18:02 purpleidea glusterbot is really JoeJulian... you might thing he's a bot, but he's actually just always on irc, and typing replies.
18:02 samppah lol
18:03 purpleidea glusterbot: thanks
18:03 glusterbot purpleidea: you're welcome
18:03 purpleidea see!
18:03 cp0k haha nice!
18:03 cp0k I am very happy I joined this channel
18:04 JMWbot joined #gluster
18:04 JMWbot I am JMWbot, I try to help remind johnmark about his todo list.
18:04 JMWbot Use: JMWbot: @remind <msg> and I will remind johnmark when I see him.
18:04 JMWbot /msg JMWbot @remind <msg> and I will remind johnmark _privately_ when I see him.
18:04 JMWbot The @list command will list all queued reminders for johnmark.
18:04 JMWbot The @about command will tell you about JMWbot.
18:04 Matthaeus1 joined #gluster
18:04 purpleidea ^^^ this bot dies sometimes... i it's back now.
18:05 failshel_ joined #gluster
18:16 primechuck JoeJulian is really a protocol droid that hangs around in IRC?  That is amazing!
18:18 recidive joined #gluster
18:19 spiekey joined #gluster
18:23 spiekey_ joined #gluster
18:23 zerick joined #gluster
18:23 _dist Ok, I'm going to venture into the wild, dangerous? world of running 30+ VMs on a gluster replicate volume. I'll make sure to document things along the way. I'm sure I'm not the first, but I think I'd be the first I've spoken to so far.
18:25 Matthaeus1 _dist, I'm doing the same thing this week.
18:25 primechuck Are you using the fuse client?
18:25 _dist no, I'll be using libgfapi
18:26 primechuck Raw libvirt or one of the managers?
18:27 _dist primechuck: Thats a LONG story, but I'm using proxmox which actually doesn't use libvirt, it uses qm and native calls to qemu
18:29 primechuck _dist:  Usually is a long story.  I haven't been using the libgfapi plugin yet, but I'm running several cloudstack installs using gluster and the fuse client and it works great.  With a few caveats.  Namely, the network-ping timeout when a brick fails
18:30 _dist primechuck: To be totally honest I haven't noticed any speed improvements from libgfapi vs fuse, but the HA is better
18:32 _dist and yes to all curious I've performed the 5-6 volume set commands RHEL recommends, they do make a big difference for libgfapi, but that difference in my expreience only brings it up to equal with fuse speeds
18:33 recidive joined #gluster
18:34 spiekey joined #gluster
18:35 japuzzo joined #gluster
18:44 JoeJulian _dist: I wish I knew what was different between your config and the myrriad of others that had much better results.
18:45 samppah _dist: have you noticed any difference in latancy libgfapi vs. fuse?
18:45 tryggvil joined #gluster
18:45 _dist JoeJulian: Well the others would have to be VM, since that's it purpose. You know anyone I could talk to about that is running it with better results?
18:46 samppah i'm quite happy with fuse performance too.. although i'm not sure what happens if all VM's start torturing IO at the same time
18:46 nage joined #gluster
18:46 nage joined #gluster
18:46 _dist samppah: Honestly didn't compare, I'm getting about 5-10ms latency on the replicate volume but I'm not running a heavy workload yet
18:46 JoeJulian _dist: They guy who wrote it. :D
18:47 samppah performance boosts from gluster 3.4.x were pretty significant
18:47 samppah for fuse
18:47 _dist JoeJulian: put me in touch :)
18:48 JoeJulian http://raobharata.wordpress.com/2012/10​/29/qemu-glusterfs-native-integration/
18:48 edong23 _dist: no real results yet?
18:48 JoeJulian Has it really been that long? Seems like 6 months ago.
18:48 _dist yeap I've read that, it's what convinced me fight for libgfapi (wich at the time meant compile it yourself
18:49 _dist edong23: Afte speaking with JoeJulian again, I've decided to go ahead as is and see what happens
18:49 _dist after*
18:49 _dist edong23: I'll device my own way of telling what is actually healing vs pretending to heal (via the heal info command)
18:50 JoeJulian Bharata follows the mailing list (or at least has in the past). That might be a good way to get in touch with him.
18:51 _dist JoeJulian: I am interested, but honestly my numbers aren't bad I'm getting about 50% of native FS
18:51 JoeJulian Plus, like I always say, don't compare apples to orchards. :D
18:52 _dist JoeJulian: Agreed, but in my case it's not hard (two single nodes, full replication) bottlenecks are easy to measure
18:52 _dist the trees are within arms reach of each other :)
18:53 JoeJulian Not saying you're having that problem, I just like the saying.
18:53 edong23 _dist: have you spun up a few VMs and simulated some busy io?
18:53 _dist yeap :) it was a good anaology
18:53 _dist edong23: yeap, I've had 20 vms running + a stress linux, nothing flatered, even with power cycling nodes
18:54 _dist edong23: JoeJulian explained that the heal info entries (the ones we think shouldn't be there) are actually a side effect, there's a bug entered for it actually.
18:54 JoeJulian bug 912239
18:54 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=912239 unspecified, unspecified, ---, vsomyaju, POST , "gluster volume heal volname info" shows files which are under modification and may not require self-heal.
18:56 _dist btw I didn't know about it, maybe just me, but stress is an awesome tool :)
18:56 JoeJulian Ironic...
18:56 JoeJulian It's often tools that cause stress.
18:56 edong23 _dist: but this doesnt solve your issue, if they always show healing... righ?
18:57 JoeJulian Like the tool that cut me off this morning...
18:57 _dist edong23: that's true, I'm going to have to find my own way around that. I had a script in mind, it sounds like JoeJulian might have a better implementation. I may pester him for help later :)
18:58 JoeJulian I'm pretty sure the python script I used is on my blog. Probably want to comment out the part where I deleted dht links though. That was for an old bug that's long gone.
18:59 _dist JoeJulian: thanks, I'm in for one more round of testing today I'll see if I can implement that. Then it's on to moving machines over (sometimes painfully) from vmware to kvm
19:00 JoeJulian That's a move that makes me happier. I'm not a fan of spending money for things that are done just as well for free.
19:01 _dist JoeJulian: I would argue in some areas done better for kvm. Not that I have many problems with VMWare.
19:01 theron joined #gluster
19:01 _dist I just _love_ the freedom of running my hypervisor on a real linux distro
19:01 JoeJulian yeah
19:02 Peanut Hi semiosis, it looks like libgfapi in qemu will make it into Ubuntu 14.04LTS? Do you have any news on that?
19:02 semiosis where did you see that? i dont have news, i want news!
19:03 _dist oh btw semiosis, did you want those compiles I did of libvirt and qemu? sorry I never followed up
19:03 Peanut https://bugs.launchpad.net/ubunt​u/+source/glusterfs/+bug/1274247
19:03 glusterbot Title: Bug #1274247 “[MIR] Glusterfs” : Bugs : “glusterfs” package : Ubuntu (at bugs.launchpad.net)
19:03 Peanut Or did I misunderstand that message?
19:03 semiosis _dist: not yet
19:04 semiosis Peanut: i saw that but idk what to make of it
19:04 semiosis i will try to get more info
19:04 Peanut Ah ok.. same here, that's why I'm asking you :-)
19:06 Peanut Also, anyone going to the Amsterdam Gluster Community meeting?
19:08 _dist joejulian: was glancing you blog for scripts, and noticed freeswitch. Is it good?
19:09 JoeJulian We've been using it since I wrote that. No issues.
19:09 edong23 JoeJulian: why didyou choose freeswitch over asterisk?
19:09 _dist joejulian: it is nice to know (haven't looked yet) that there are solid things out there. I can not even, maybe you know, how bad some of the proprietary stuff is
19:09 edong23 i know many of the differences (mostly in  signalling)
19:10 JoeJulian I was pegging one cpu core on asterisk with some mild testing. Never even come close with FS (which is properly multithreaded).
19:11 * semiosis uses onsip, FTW
19:11 edong23 you were transcoding?
19:12 JoeJulian That was just switching and ivr with 100 calls.
19:12 semiosis i dont usually plug commercial stuff, but the people at onsip are top notch!
19:12 semiosis if you're ever looking for hosted pbx
19:14 johnmark semiosis: nice - will have to look into that
19:21 _dist JoeJulian: this script right? http://joejulian.name/blog/quick-and-d​irty-python-script-to-check-the-dirty-​status-of-files-in-a-glusterfs-brick/ you're saying comment out the os.remove stuf?)
19:21 glusterbot Title: Quick and dirty python script to check the dirty status of files in a GlusterFS brick (at joejulian.name)
19:21 JoeJulian right
19:21 _dist great, I'll include it in my testing today
19:28 matclayton joined #gluster
19:29 theron joined #gluster
19:35 diegows joined #gluster
19:41 primechuck _dist:  Yeah.  The fuse client performs quite well for our cloudstack environments (and libgfapi isn't natively supported in cloudstack yet) the only thing I've found with the fuse client it consumes about a core worth of CPU time servicing interrupts, but that is an easy problem to live with.
19:51 ninkotech__ joined #gluster
19:52 gdubreui joined #gluster
19:54 spiekey joined #gluster
19:55 ninkotech__ joined #gluster
19:55 jmarley joined #gluster
19:58 B21956 joined #gluster
20:02 ninkotech__ joined #gluster
20:02 andreask joined #gluster
20:09 ninkotech__ joined #gluster
20:11 purpleidea in case anyone knows: is it true that RHEL/CentOS7 won't have python3 packages available?
20:14 ctria joined #gluster
20:16 JoeJulian True and false.
20:17 JoeJulian python3 should be in epel.
20:18 Oneiroi joined #gluster
20:18 gdubreui joined #gluster
20:19 spiekey joined #gluster
20:22 purpleidea JoeJulian: I don't quite see it: https://dl.fedoraproject.org/pub/epel/​6/x86_64/repoview/letter_p.group.html
20:22 glusterbot Title: RepoView: "Fedora EPEL 6 - x86_64" (at dl.fedoraproject.org)
20:23 purpleidea same here: https://dl.fedoraproject.o​rg/pub/epel/beta/7/x86_64/
20:23 glusterbot Title: Index of /pub/epel/beta/7/x86_64 (at dl.fedoraproject.org)
20:27 ninkotech joined #gluster
20:28 LessSeen_ just want an expert opinion on this if possible - i am totally new to gluster, so i may not have this test set up in an optimal way.  i have 3 lamp servers in 3 different ec2 regions that are sharing a replicated glusterfs volume for apache data.  there is one brick per server (set to replicate 3), each mounts the gluster vol from 127.0.0.1 and the replicated filesystem seems to work fine, it is just slow.  any ideas
20:28 LessSeen_ how i should have this setup to retain multi-region redundancy, but get better performance?
20:30 JoeJulian purpleidea: Ah, I guess that's right. Stephen Smoogen has stated that although he won't impede anyone who wants to package python 3 for EPEL, he's not going to be maintaining it since it (apparently) breaks compatibility with every minor release. He is not going to maintain a python3.4, python 3.5, etc rpm tree.
20:31 purpleidea JoeJulian: okay, so maybe yes, maybe no. sounds good, thanks. too bad actually... would like to use some python3 things, but not if i can't guarantee they exist in centos7.
20:32 khushildep joined #gluster
20:32 ninkotech joined #gluster
20:38 ninkotech joined #gluster
20:46 nage joined #gluster
20:46 nage joined #gluster
20:52 natgeorg joined #gluster
20:52 natgeorg joined #gluster
20:53 ninkotech joined #gluster
21:05 failshell joined #gluster
21:07 ninkotech_ joined #gluster
21:08 Matthaeus joined #gluster
21:09 cp0k hmm, in Gluster 3.4.2 when I do '#  gluster volume heal th-tube-storage info' it is still not returning anything
21:09 cp0k I dont even get the exit code
21:09 cp0k oh wait, finally got one "146"
21:11 LessSeen joined #gluster
21:12 cp0k any idea why I would get this?
21:15 JoeJulian 146 is ECONNREFUSED
21:16 cp0k yes, just found that as well here - https://bugzilla.redhat.co​m/show_bug.cgi?id=1019908
21:16 glusterbot Bug 1019908: urgent, high, RHS 2.1.2, vmallika, VERIFIED , Error while executing action CommitRemoveGlusterVolumeBricks: Command execution failed return code: 146
21:16 JoeJulian Wait... no it's not...
21:16 JoeJulian At least not in Linux... That's 111.
21:16 cp0k I am running Gentoo Linux
21:16 JoeJulian Looks like it's 146 in solaris.
21:17 ninkotech__ joined #gluster
21:19 Matthaeus OpenVZ on top of gluster 3.4.  Am I nuts?
21:20 lpabon joined #gluster
21:20 JoeJulian Matthaeus: From what I've read on various internet sites you are...
21:20 JoeJulian Matthaeus: Oh, you mean about that...
21:21 Matthaeus :P
21:21 JoeJulian Matthaeus: Expect issues, but they all should be solvable. I've not seen anyone document it though.
21:21 Matthaeus Yay!
21:21 Matthaeus _dist, you around?
21:21 plarsen joined #gluster
21:22 Matthaeus If you're storing KVM disk images on gluster, make sure you're using writeback caching.
21:24 _dist I am
21:24 _dist no I am using raw
21:24 _dist (I want live migration)
21:25 ninkotech joined #gluster
21:26 Matthaeus Qcow2 doesn't preclude live migration.
21:26 LessSeen the silence speaks volumes jk* thanks anyway guys
21:26 _dist Mattheaus: oh I'm sorry that's not what I mean to say. I meant to say I'm either using qcow2 or raw, with cache=none (writeback/writethrough are bad for migration)
21:27 Matthaeus s/writeback/writethrough/g
21:27 glusterbot Matthaeus: Error: u"s/writeback/writethrough/g If you're storing KVM disk images on gluster, make sure you're using writeback caching." is not a valid regular expression.
21:28 Matthaeus That's odd.  When I try to boot a qcow2 image with no cache, I get errors on my setup.
21:28 _dist Matthaeus what' the error?
21:29 Matthaeus http://pastie.org/8723770
21:29 glusterbot Title: #8723770 - Pastie (at pastie.org)
21:30 JoeJulian Heh, looks like glusterbot doesn't like parenthesis.
21:30 recidive joined #gluster
21:30 _dist Matthaeus: what filesystem do you have under gluster?
21:30 Matthaeus ext4
21:31 _dist Matthaeus: unless I'm missing something, Is the only error there "failed exit code 1" ?
21:31 Matthaeus At the top:  could not open disk image /mnt/data/images/200/vm-200-disk-1.qcow2: Invalid argument
21:32 Matthaeus There are two lines there; pastie just word-wraps a lot.
21:32 Matthaeus Enabling write-through caching makes this problem magically go away.
21:33 _dist Matthaeus: I'd try a manual launch with with verbose setting to get full errors
21:34 _dist Matthaeus: as far as I know there is no good reason you'd have to use emulated disk cache on ext4 and glusterfs unless you mounted with fuse using direct-io-mode disabled
21:35 Matthaeus Shutting down the VM now to try a manual start.
21:39 Matthaeus Neither qm nor the underlying kvm have a useful —verbose option.  Syslogs show the exact same error.
21:39 Matthaeus I am mounted with fuse.
21:40 _dist can you show me your mount output ? I've definitely ran with cache=none on gluster before
21:40 _dist Matthaeus: I also noticed you're using kvm binary, how old is your qemu?
21:41 Matthaeus http://pastie.org/8723796
21:41 glusterbot Title: #8723796 - Pastie (at pastie.org)
21:41 Matthaeus It's still kernel 2.6.  I'm running proxmox, and they backport a lot of stuff into 2.6, but it's still 2.6.
21:42 _dist Matthaeus: Ah I'm also using proxmox, yeah they just rename their qemu-system-x86_64 to kvm for some reason
21:43 _dist Matthaeus: what version of proxmox? 3.1 actually uses libgfapi, just uses fuse mount for browsing and uploading ISOs etc
21:46 _dist Matthaeus: did you intentionally avoid that by mapping the mount.glusterfs manually and then pointing proxmox to a DIR for the image files?
21:47 ninkotech_ joined #gluster
21:48 Matthaeus _dist:  I did, yes.
21:48 Matthaeus Proxmox 3.1's gluster support is fantastically broken.
21:48 Matthaeus It supports images only, not backups, not isos.
21:50 Matthaeus Half the reason I'm using glusterfs is so that my backups are resilient.
21:50 Matthaeus But I suppose I could use native for vm images, and then use this manual mount for backups and isos.
21:52 matclayton joined #gluster
21:53 kkeithley_ 3.5.0beta3 RPMs for Fedora, RHEL, and CentOS are up in YUM repo at http://download.gluster.org/pub/glust​er/glusterfs/qa-releases/3.5.0beta3/
21:53 glusterbot Title: Index of /pub/gluster/glusterfs/qa-releases/3.5.0beta3 (at download.gluster.org)
21:57 theron joined #gluster
22:01 theron joined #gluster
22:01 _dist Matthaeus: I use ISOs ,images, migration in proxmox over libgfapi without issue so far (yeah you can't upload files through it that's true)
22:02 plarsen joined #gluster
22:03 jclift_ JoeJulian: Thanks for the NFS help on gluster-users.  I have 0 clue with the older gluster, and was really struggling to figure out how to help the guy.
22:04 JoeJulian No problem. As soon as I saw which version he was running, I realized where you were going to get stuck.
22:04 jclift_ :)
22:10 Matthaeus Gah, somehow I ended up with split brain on my test vm image.
22:10 _dist Mattheaus: did you take a node down, bring up, and then the other before heal finished? Or did you access it directly (not through a mount?)
22:11 Matthaeus Probably the first.
22:11 Matthaeus I've been testing STONITH for unrelated purposes.
22:12 in joined #gluster
22:13 Matthaeus My fencing tests are complete, though, so I should just nuke the VM and start over.
22:16 Matthaeus Ah, this is working much better.
22:16 _dist Matthaeus: full disclosure, I'm using the pvetest repos, I just don't think it's that dangerous personally :)
22:16 _dist Matthaeus: but I was using the old one just yesterday
22:17 Matthaeus These are going to be production boxen once I get them through the shakedown phase.  I won't have easy physical access to them, so I'm going with the 3.1 release.
22:17 _dist yeah I understand, I just take risks I guess :) I wanted new qemu and gluster 3.4.2 instead of 3.4.1
22:18 Matthaeus Is the new kvm_intel module in that branch?
22:18 Matthaeus I want nested KVM so badly I can taste it when I fall asleep at night.
22:21 _dist let me check
22:22 _dist what the hell! it totally isn't that makes no sense!, I should probably calm down and put it in the modprobe first, but normally there's a nested that contains 0 by default
22:23 ninkotech joined #gluster
22:23 _dist I actually had vmware running inside of kvm working was able to live migrate it (that wasn't proxmox though). I think proxmox is pretty decent for pro use, home use I use libvirt
22:24 Matthaeus I want to start playing around with openstack, but I don't have the hardware at home to run it in more than a sandbox.
22:25 JoeJulian That's forever the problem.
22:28 recidive joined #gluster
22:28 allig8r joined #gluster
22:32 jobewan joined #gluster
22:34 ninkotech_ joined #gluster
22:37 Matthaeus Fortunately, it's not that hard to justify having a few screaming 1U servers in my office, as long as it's temporary.
22:37 Matthaeus My wife complains when I put them in the living room, though.
22:38 JoeJulian I have a rack in a "server room" for that. It also houses the furnace and the hot water heater.
22:38 Matthaeus I live in half a house in San Francisco.  I'm sure there's a "server room" somewhere in the house, but not in the portion I'm paying for.
22:39 Matthaeus And I also don't want to be paying SF power rates for a small server farm.
22:39 Matthaeus So these machines are going to Nevada this weekend, where the power is cheap and the data retention laws are lax.
22:39 JoeJulian Ew...yeah...
22:39 semiosis there was someone who used to come around here named Matthaeus, i thought he was in montana though...
22:39 semiosis is that you?
22:39 semiosis same person?
22:39 Matthaeus Same person.
22:39 semiosis welcome back
22:39 Matthaeus Thank you!
22:40 JoeJulian You left? ;)
22:40 semiosis i'll be in bozeman briefly in may
22:40 semiosis for like an hour lol
22:40 Matthaeus Left the job at Montana.edu, wasn't using gluster anywhere for a while.
22:40 Matthaeus Spent a year making video games, which required no gluster at all.
22:41 JoeJulian What're you doing in Bozeman for an hour?
22:41 semiosis passing through on my way to ennis to visit family
22:42 JoeJulian My great great grandfather got the Mayor of Bozeman arrested once...
22:42 semiosis cool story bro ;)
22:42 JoeJulian Hehe, he was serving wine in his restaurant during prohibition.
22:42 semiosis wow!
22:43 ninkotech_ joined #gluster
22:49 semiosis Matthaeus: you know there's going to be gluster things happening in april in SF, right?
22:49 Matthaeus I didn't, actually.
22:49 Matthaeus I've only just now dipped my toe back in.
22:50 semiosis idk exactly what yet, but the red hat summit & devnation are happening, april 14 - 17 at moscone ctr
22:50 JoeJulian johnmark: Are we invited to that?
22:50 semiosis i'm sure (and I hope) there will be community events as well outside the main convention
22:52 Matthaeus I work at Splunk, which is easy walking distance to there.
22:52 semiosis neat
22:55 theron joined #gluster
23:02 gdubreui joined #gluster
23:04 KyleG joined #gluster
23:04 KyleG joined #gluster
23:06 Matthaeus awesome.  Gluster native with qcow2 supports live snapshots.
23:06 Matthaeus Let's teset migration now.
23:07 ninkotech_ joined #gluster
23:07 Matthaeus And migration works perfectly.
23:08 Matthaeus Awesome.  I'm now a happy person.
23:09 _dist yeap me too
23:10 _dist all out of the box too, takes me like 40 min (practiced) to get that on say ubuntu. A whole day the first time
23:11 _dist Matthaeus: also (I know this is a gluster channel) but proxmox pve test has a ceph GUI in the client, not just for mounts but for creating and managing OSDs (might give that a try on the side) but my intention is to go with gluster if/until something crazy happens.
23:11 Matthaeus Ceph and other file systems that require a central metadata server scare me.
23:12 ninkotech__ joined #gluster
23:14 _dist Matthaeus: I'd agree, but there's a pveceph install command that sets it all up on your proxmox nodes for you, and distributes the meta data across them both if you tell it (as far as I read). Haven't tried any of it yet
23:15 Matthaeus _dist:  your ideas intrigue me...
23:16 _dist Matthaeus: something I've yet to do, is get proxmox to migrate over my SAN (it uses vmbr0 by default). Not sure what you have, but I've got a dedicated network for gluster traffic
23:17 Matthaeus Each machine has two PHYs, so I'm sending gluster/corosync/vmbr1 all over a private link, and reserving the other one for public internet access/STONITH
23:19 Matthaeus I'd love to add a quad-interface card to each one, but I've already spent about $500 on this project, and that's a lot for something that's not going to bring in any money.
23:20 ninkotech__ joined #gluster
23:20 _dist Matthaeus: how did you do that? I just used the default install. I found startek quad cards to be decent (for port quantity) and they are < $100
23:20 ktosiek joined #gluster
23:21 Matthaeus _dist: playing around with /etc/network/interfaces to add the extra vmbr1, then edit /etc/hosts so that the machines think of vmbr1 as the interface that proxmox runs on.  Then join them in a cluster and you're there.
23:22 _dist but then what about web access?
23:22 Matthaeus Same thing with gluster.  Each server only knows about the other by its private ip.
23:22 Matthaeus Proxmox runs on 0.0.0.0:8443, so that's not a problem.
23:22 _dist Matthaeus: yeah I did that for gluster, oh it does? Ah, so if I just perform my pvecm node stuff on my san IPs I'll get what I want
23:22 _dist Honestly migration on my lan doesn't take long, I just think it's dirty
23:23 StarBeast joined #gluster
23:23 Matthaeus Yeah.
23:23 Matthaeus And I think separating migration from corosync would be a lot more involved.
23:23 _dist Matthaesu: Also, I did this http://boffblog.wordpress.com/2013/08/22/how​-to-install-proxmox-ve-3-0-on-software-raid/ (he's got a few syntax errors in it, but the idea is sound).
23:23 theron joined #gluster
23:24 Matthaeus My bricks are software raid backed.
23:24 Matthaeus The OS isn't, but the OS is meant to be expendable if necessary.
23:24 Matthaeus That is, after all, why I have two of these machines.
23:25 _dist Matthaeus: I guess now (only got a few VMs) is better than later to tear down my cluster later, more of a pain later. Yeah mine is software raid backed too, but I wanted my OS to be as well so that's how I did it
23:26 dkorzhevin joined #gluster
23:28 _dist Matthaeus: crap, looks like migration might be broke in pvetest
23:29 Matthaeus Boo hiss!
23:29 Matthaeus _dist: how many machines are you running?
23:29 _dist Matthaeus: just one test right now, 30-32 before next week
23:30 Matthaeus phsycial, not virtual
23:30 _dist oh, 3, but 2 in test mode right now
23:30 Matthaeus ok
23:33 _dist Matthaeus: Look slike migration is optimized for thin, maybe through virtio baloon driver. It threw me when it was going like 800MB/s over 1GBs but it doesn't appear broken, just wrong on average speed
23:41 _dist well, it's home time! I'll let you guys know how everything goes tommorrow
23:42 _dist left #gluster
23:45 eryc joined #gluster
23:45 KyleG1 joined #gluster
23:49 sprachgenerator joined #gluster
23:51 eryc joined #gluster
23:51 eryc joined #gluster
23:58 recidive joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary