Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-01-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:14 partner Teela: maybe read http://joejulian.name/blog/gluster​fs-split-brain-recovery-made-easy/ to learn about split-brain and how to recover from that, haven't tried the tool myself
00:14 partner its past 2AM so i'm heading to bed now ->
00:40 ilde joined #gluster
01:06 bala joined #gluster
01:07 psi_ joined #gluster
01:38 tryggvil_ joined #gluster
01:41 nangthang joined #gluster
01:45 Gill joined #gluster
01:58 plarsen joined #gluster
01:58 plarsen joined #gluster
02:14 haomaiwa_ joined #gluster
02:17 julim joined #gluster
02:19 harish joined #gluster
02:21 MugginsM joined #gluster
02:26 badone joined #gluster
02:26 plarsen joined #gluster
02:27 hagarth joined #gluster
02:31 bharata-rao joined #gluster
02:40 msmith_ joined #gluster
02:48 plarsen joined #gluster
03:01 thangnn_ joined #gluster
03:18 nishanth joined #gluster
03:22 fandi joined #gluster
03:26 dgandhi joined #gluster
03:27 dgandhi joined #gluster
03:28 dgandhi joined #gluster
03:29 dgandhi joined #gluster
03:30 dgandhi joined #gluster
03:31 dgandhi joined #gluster
03:33 dgandhi joined #gluster
03:34 dgandhi joined #gluster
03:36 dgandhi joined #gluster
03:37 dgandhi joined #gluster
03:38 dgandhi joined #gluster
03:39 dgandhi joined #gluster
03:40 dgandhi joined #gluster
03:42 dgandhi joined #gluster
03:43 itisravi joined #gluster
03:44 dgandhi joined #gluster
03:45 hagarth joined #gluster
03:45 dgandhi joined #gluster
03:48 dgandhi joined #gluster
03:48 MugginsM joined #gluster
03:49 dgandhi joined #gluster
03:50 dgandhi joined #gluster
03:51 dgandhi joined #gluster
03:52 dgandhi joined #gluster
03:54 dgandhi joined #gluster
03:56 dgandhi joined #gluster
03:56 dgandhi joined #gluster
03:58 dgandhi joined #gluster
03:59 kanagaraj joined #gluster
04:00 nbalacha joined #gluster
04:00 dgandhi joined #gluster
04:01 dgandhi joined #gluster
04:02 Rapture joined #gluster
04:03 nbalacha joined #gluster
04:03 dgandhi joined #gluster
04:04 dgandhi joined #gluster
04:05 dgandhi joined #gluster
04:07 dgandhi joined #gluster
04:10 dgandhi joined #gluster
04:11 dgandhi joined #gluster
04:12 dgandhi joined #gluster
04:13 hagarth1 joined #gluster
04:14 dgandhi joined #gluster
04:15 dgandhi joined #gluster
04:17 dgandhi joined #gluster
04:18 dgandhi joined #gluster
04:19 dgandhi joined #gluster
04:20 atinmu joined #gluster
04:20 dgandhi joined #gluster
04:22 dgandhi joined #gluster
04:23 dgandhi joined #gluster
04:25 dgandhi joined #gluster
04:27 dgandhi joined #gluster
04:27 RameshN joined #gluster
04:28 dgandhi joined #gluster
04:30 dgandhi joined #gluster
04:32 dgandhi joined #gluster
04:33 dgandhi joined #gluster
04:35 dgandhi joined #gluster
04:36 dgandhi joined #gluster
04:38 dgandhi joined #gluster
04:38 dgandhi joined #gluster
04:39 shubhendu joined #gluster
04:40 dgandhi joined #gluster
04:41 dgandhi joined #gluster
04:42 spandit joined #gluster
04:43 dgandhi joined #gluster
04:44 Teela is there anyway to artificially create splitbrain. I'm trying to test the full heal on our dev system
04:45 anil joined #gluster
04:47 ndarshan joined #gluster
04:48 atinmu Teela, itisravi can answer u
04:49 Teela is he around?
04:51 itisravi Teela: you can turn self-heal off and bring down replica bricks alternatively when writing to the file from the mount point.
04:52 Teela how do i do that?
04:52 Teela im a bit of a newbie with gluster
04:52 Teela thanks for your help
04:52 Teela btw
04:52 itisravi Teela: btw, split-brained files need manual intervention to heal. See https://github.com/GlusterFS/glusterfs/b​lob/master/doc/debugging/split-brain.md
04:52 Teela well
04:52 Teela my plan
04:52 Teela is to stop
04:53 Teela one node
04:53 Teela clear the brick
04:53 Teela and do a heal the full volume
04:53 Teela to remedy the the split brain issues
04:54 rjoseph|afk joined #gluster
04:56 hagarth joined #gluster
04:57 Teela will that work?
04:57 Teela :)
05:02 itisravi Teela: 1) Create and start a 1x2 volume, 2)Mount it, and create a file. 3)bring down node 1 and write to the file from the mount, 4)bring dowm node 2, bring back node 1 and write to the file from mount, 5) bring back node 2 as well.
05:03 rafi joined #gluster
05:03 itisravi Now you have 2 nodes having same file but different content. This is a split-brain. When you try to access such a file from the mount, you would get an input/output error.
05:05 anoopcs joined #gluster
05:10 nshaikh joined #gluster
05:20 kdhananjay joined #gluster
05:22 ppai joined #gluster
05:24 jiffin joined #gluster
05:27 lpabon joined #gluster
05:27 gem joined #gluster
05:31 nbalachandran_ joined #gluster
05:32 kshlm joined #gluster
05:33 Manikandan joined #gluster
05:37 kanagaraj joined #gluster
05:38 lalatenduM joined #gluster
05:44 bala joined #gluster
05:45 atinmu joined #gluster
05:45 hchiramm_ joined #gluster
05:46 dusmant joined #gluster
05:51 pp joined #gluster
06:00 m0zes joined #gluster
06:01 smohan joined #gluster
06:03 soumya_ joined #gluster
06:12 glusterbot News from newglusterbugs: [Bug 1166020] self-heal-algorithm with option "full" doesn't heal sparse files correctly <https://bugzilla.redhat.co​m/show_bug.cgi?id=1166020>
06:12 sakshi joined #gluster
06:13 overclk joined #gluster
06:22 meghanam joined #gluster
06:35 atalur joined #gluster
06:39 suman_d joined #gluster
06:45 fandi joined #gluster
06:46 anrao joined #gluster
06:46 anrao gem,
06:47 gem anrao, yep
06:47 anrao is this the one ? gem
06:51 a2 joined #gluster
06:53 karnan joined #gluster
07:02 mbukatov joined #gluster
07:08 sputnik13 joined #gluster
07:09 atinmu joined #gluster
07:10 jbrooks joined #gluster
07:11 hagarth joined #gluster
07:20 jtux joined #gluster
07:22 dusmant joined #gluster
07:23 sputnik13 joined #gluster
07:29 dusmant joined #gluster
07:30 fandi joined #gluster
07:32 atalur joined #gluster
07:42 glusterbot News from newglusterbugs: [Bug 1161416] snapshot delete all command fails with --xml option. <https://bugzilla.redhat.co​m/show_bug.cgi?id=1161416>
07:45 abyss^ someone here?
07:47 Fen1 joined #gluster
07:53 dusmant joined #gluster
07:53 anoopcs joined #gluster
07:56 jvandewege__ joined #gluster
08:01 rjoseph|afk joined #gluster
08:03 atalur joined #gluster
08:11 raghu joined #gluster
08:14 fandi joined #gluster
08:17 [Enrico] joined #gluster
08:23 fandi joined #gluster
08:35 ricky-ticky1 joined #gluster
08:35 anil joined #gluster
08:42 Mortem joined #gluster
08:42 Mortem Hello
08:42 glusterbot Mortem: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
08:43 Mortem I am running a new gluster installation using 3.6.1
08:43 Mortem All looks fine
08:43 fsimonce joined #gluster
08:43 fsimonce joined #gluster
08:44 Mortem I setup geo-replication as requested using creating passwordless authentication between servers
08:44 Mortem I can create successfully the georeplication
08:44 Mortem but when I start it, it is faulty
08:44 Mortem in logs I have this: RepceServer terminating on reaching EOF
08:45 Mortem and "worker died in startup phase"
08:45 Mortem Any idea would be helpful
08:45 Mortem thanks
08:51 karnan joined #gluster
08:58 kanagaraj joined #gluster
08:59 meghanam joined #gluster
09:03 atalur joined #gluster
09:04 Intensity joined #gluster
09:05 kdhananjay joined #gluster
09:11 hybrid512 joined #gluster
09:13 harish_ joined #gluster
09:15 jvandewege_ joined #gluster
09:22 dusmant joined #gluster
09:24 harish joined #gluster
09:25 gvandeweyer joined #gluster
09:27 gvandeweyer hi all, we're considering to use/activate gluster on our local cluster to streamline data access. Is it feasible to have 2 big nodes (>15T), together with several smaller (<5Tb) nodes, or will this give problems ?
09:28 gvandeweyer Also, is there a "raid5" type solution yet? I read something about erasure coding on blogs, but couldn't find anything but simple replicate on the docs.
09:28 kanarip gvandeweyer, a single node could serve multiple bricks
09:28 gvandeweyer kanarip: it would be better to create multiple similar sized bricks than 1 big brick for the big nodes?
09:29 kanarip i would think, but i'm not by far the most experienced around here
09:29 kanarip in fact, i'm kinda new here ;-) so don't take my world for it
09:30 gvandeweyer right, no problem
09:30 kanarip word*
09:32 Manikandan joined #gluster
09:33 rafi1 joined #gluster
09:36 atalur joined #gluster
09:42 deniszh joined #gluster
09:45 T0aD joined #gluster
09:46 sputnik13 joined #gluster
09:53 ndevos gvandeweyer: yes, that is possible
09:54 sputnik13 joined #gluster
09:54 ndevos in general we advise to use equal sized bricks, but with 3.6 you can also mix brick sizes
09:55 ndevos before, smaller bricks would fill up quicker than the large bricks, that is not the case anymore - the size of the bricks is now taken into consideration on where to place the data
09:57 kanagaraj joined #gluster
09:59 ndevos hmm, well, in fact, I wonder if that made it in 3.6... the planning page lists it, but the its feature page says it was abandoned?!
09:59 ndevos http://www.gluster.org/community/d​ocumentation/index.php/Planning36
10:00 ndevos raghu: do you know if the "different sized feature" made it into 3.6?
10:01 shubhendu joined #gluster
10:02 ndevos gvandeweyer: in general, use RAID 6/10 for the bricks, and have raid sets of ~12 disks, that seems to be a common recommendation
10:05 kalzz joined #gluster
10:08 gvandeweyer ndevos : ok, thanks for that information.
10:09 sputnik13 joined #gluster
10:11 kdhananjay joined #gluster
10:12 meghanam joined #gluster
10:12 ppai joined #gluster
10:16 overclk joined #gluster
10:16 anrao joined #gluster
10:18 nbalachandran_ joined #gluster
10:24 rrottach joined #gluster
10:31 rafi joined #gluster
10:35 siel joined #gluster
10:40 ctria joined #gluster
10:43 glusterbot News from newglusterbugs: [Bug 1127457] Setting security.* xattrs fails <https://bugzilla.redhat.co​m/show_bug.cgi?id=1127457>
10:48 ctria joined #gluster
10:49 bharata-rao joined #gluster
10:57 anrao joined #gluster
10:57 rgustafs joined #gluster
11:01 kanagaraj joined #gluster
11:07 Philambdo joined #gluster
11:13 ppai joined #gluster
11:14 dusmant joined #gluster
11:22 meghanam joined #gluster
11:29 Philambdo joined #gluster
11:37 shubhendu joined #gluster
11:37 RameshN joined #gluster
11:41 rjoseph|afk joined #gluster
11:43 glusterbot News from newglusterbugs: [Bug 1174170] Glusterfs outputs a lot of warnings and errors when quota is enabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1174170>
11:43 glusterbot News from newglusterbugs: [Bug 1174250] Glusterfs outputs a lot of warnings and errors when quota is enabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1174250>
11:43 glusterbot News from resolvedglusterbugs: [Bug 1174247] Glusterfs outputs a lot of warnings and errors when quota is enabled <https://bugzilla.redhat.co​m/show_bug.cgi?id=1174247>
11:53 partner gvandeweyer: what i've done is i've set up the min-disk-free option for the volume so efficiently it will stop writing to the bricks when that limit is met and writes go to bricks with more space
11:54 partner i'm basically just adding more servers to the cluster and most of the writes go there, the old ones are "full"
11:58 hagarth joined #gluster
12:13 rafi1 joined #gluster
12:19 nangthang joined #gluster
12:24 calum_ joined #gluster
12:30 ira joined #gluster
12:33 ira joined #gluster
12:40 RicardoSSP joined #gluster
12:40 RicardoSSP joined #gluster
12:40 peem Is there any sort of GUI to gluster ? I have found oVirt, but it seems to be part of bigger system and not usable only for gluster really. gluster-deploy.py seems noice, but gives me an error and does not do much then. Anything else there ?
12:41 Norky joined #gluster
12:43 overclk joined #gluster
12:45 overclk_ joined #gluster
12:45 LebedevRI joined #gluster
12:54 harish joined #gluster
12:56 badone joined #gluster
12:59 atalur joined #gluster
12:59 Fen1 joined #gluster
13:04 plarsen joined #gluster
13:13 glusterbot News from newglusterbugs: [Bug 1185322] Enabling SSL for glusterd will cause all nodes to crash when connection is interrupted <https://bugzilla.redhat.co​m/show_bug.cgi?id=1185322>
13:19 kanagaraj_ joined #gluster
13:30 B21956 joined #gluster
13:31 _Bryan_ joined #gluster
13:32 Slashman joined #gluster
13:42 partner peem: not that i'm aware of. RH is building something up but that's years away. there is some dashboard but i think its one-way only, haven't tried it out yet
13:42 partner not sure if you are looking something for configuring or monitoring / viewing any status
13:43 partner peem: anyways here's one dash: http://aravindavk.in/blog/introducing-gdash/
13:43 Gill joined #gluster
13:54 sputnik13 joined #gluster
13:59 rgustafs joined #gluster
14:02 rjoseph|afk joined #gluster
14:11 dgandhi joined #gluster
14:12 jvandewege joined #gluster
14:17 jmarley joined #gluster
14:22 bene2 joined #gluster
14:23 elico joined #gluster
14:33 bala joined #gluster
14:34 jvandewege_ joined #gluster
14:35 doubt joined #gluster
14:39 neofob joined #gluster
14:51 peem partner: thanks. I will try if ovirt does not work for my need.
14:55 kanagaraj joined #gluster
14:56 booly-yam-1010 joined #gluster
14:57 _Bryan_ joined #gluster
14:57 soumya_ joined #gluster
15:03 virusuy joined #gluster
15:09 vikumar joined #gluster
15:10 wushudoin joined #gluster
15:23 rwheeler joined #gluster
15:24 sputnik13 joined #gluster
15:37 bennyturns joined #gluster
15:37 bala joined #gluster
15:41 [Enrico] joined #gluster
15:41 ProT-0-TypE joined #gluster
15:44 monotek1 joined #gluster
15:56 bala joined #gluster
16:02 lmickh joined #gluster
16:08 booly-yam-9299 joined #gluster
16:09 tdasilva joined #gluster
16:11 ralala joined #gluster
16:12 jbrooks joined #gluster
16:13 daMaestro joined #gluster
16:14 ralala joined #gluster
16:16 calisto joined #gluster
16:17 soumya joined #gluster
16:20 daMaestro|isBack joined #gluster
16:24 ProT-O-TypE joined #gluster
16:27 ProT-O-TypE joined #gluster
16:29 ProT-0-TypE joined #gluster
16:34 wkf joined #gluster
16:36 timbyr_ joined #gluster
16:47 anti[Enrico] joined #gluster
16:59 T3 joined #gluster
17:00 T3 hey guys, I'm getting a series of messages like this on my gluster.log:
17:00 T3 [2015-01-23 16:58:45.989486] W [client-rpc-fops.c:1817:client3_3_fxattrop_cbk] 0-site-images-client-1: remote operation failed: Transport endpoint is not connected
17:00 T3 it happens, than stop happening, then comes back
17:00 T3 I'm been watching that for days
17:00 T3 and googling
17:01 T3 rsync are not the guild here (at least alone) because I have observed it without rsync running
17:01 T3 the last finding says it may be ext4 fault, pointing to ifs
17:01 T3 xfs*
17:01 T3 http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
17:02 T3 this post is 2.5 years young. is it still valid?
17:11 Fen1 joined #gluster
17:14 Teela Hello
17:14 glusterbot Teela: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
17:15 Teela I am trying to figure out how to artifically create a splitbrain errors for my dev environment to play around with some self heal scripts
17:15 Teela anyone know how i may replicate the split brains
17:15 Teela please and thank you
17:18 Psi|4ward left #gluster
17:18 squizzi joined #gluster
17:28 T3 Teela, have you seem https://github.com/joejulian/glusterfs-splitbrain ?
17:29 T3 Teela, http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/
17:42 DJClean joined #gluster
17:47 hagarth joined #gluster
17:47 booly-yam-9299_ joined #gluster
17:50 MacWinner joined #gluster
17:55 dbruhn joined #gluster
17:59 shubhendu joined #gluster
18:05 MarcinEF joined #gluster
18:07 vikumar joined #gluster
18:09 B21956 joined #gluster
18:10 T3 I just found the "A kernel change breaks GlusterFS" post at https://lwn.net/Articles/544298/
18:11 T3 What exactly is the behavior expected when one use GlusterFS and ext4 nowadays?
18:11 T3 I mean, what kind of issues would one notice?
18:15 partner Teela: you have given all the instructions to reproduce that, please follow them
18:16 partner itisravi gave 5 steps to follow which are very simple
18:20 partner T3: while i'm not sure, those are ancient messages and for example gluster has evolved several majors since, i haven't heard anybody complaining about etx4 for a long time
18:21 partner T3: but, maybe wait till some ext4 users appear or anybody with further knowledge, can't track the bugs right now
18:26 Teela I looked at those links and they dont talk about how to replicate a splitbrain only how to fix them :)
18:26 partner Teela: you were given instructions on this irc channel to produce split-brain by itisravi
18:27 Teela really, maybe i missed them I will scroll up
18:27 partner 07:02 < itisravi> Teela: 1) Create and start a 1x2 volume, 2)Mount it, and create a file. 3)bring down node 1 and write to the file from the mount, 4)bring dowm node 2, bring back node 1 and write to the file from mount, 5) bring back node 2 as well.
18:27 partner 07:03 < itisravi> Now you have 2 nodes having same file but different content. This is a split-brain. When you try to access such a file from the mount, you would get an input/output error.
18:28 partner timezone of course local to mine.. but that's how to do it
18:28 Teela oh i did miss it
18:28 Teela THANK YOU
18:28 _dist joined #gluster
18:28 Teela thanks a lot guys
18:30 partner np
18:30 lalatenduM joined #gluster
18:31 partner but if you think that requires you to make a split-brain then what exactly caused your split-brain situation.. ?? something to think and perhaps something to put effort to prevent it from happening again
18:31 partner over 1000 files is a quite much and the gluster just cannot know which version is the proper one
18:36 Teela yes
18:36 Teela so i tried those instruction
18:36 Teela doesnt seem to make split brain
18:37 Teela and how im bringing the node downi s just by stopping glusterfs
18:39 dbruhn An easier way to create a split brain issue, is to take a brick server offline, and then once the brick server is down write the data to the system, and after you've done that, hard power the client that is doing the write.
18:39 Bardack joined #gluster
18:41 dbruhn The fuse client is aware of what is supposed to be written to a replicant pair, and if the mount is still up and active it will wait and write it once the brick comes back online.
18:41 JoeJulian ?
18:41 JoeJulian That shouldn't create a split-brain.
18:41 dbruhn Am I wrong on that?
18:42 dbruhn If a brick server is down while the system is being written to, and the client goes offline as well after new data is written to the other brick?
18:42 dbruhn At least in 3.3 I was able to create split-brain issues that way
18:42 JoeJulian Two ways of doing it: 1) two clients, one on each server. Disconnect the network cable. Edit the same file on both clients. Reconnect the network cable.
18:43 JoeJulian 2) Down server1. Edit a file. Down server2, bring server1 back. Edit the same file. Bring server2 back.
18:44 JoeJulian The first is classic network partition split-brain. The second we coined split-brain-over-time.
18:50 JoeJulian Heh, and I just scrolled back and read partner's explanation that could have saved me some typing.
18:51 JoeJulian T3: ext4, with any current kernel and gluster version, should be fine (if you like bloated messy filesystem code, imho).
18:51 daMaestro joined #gluster
18:53 JoeJulian Teela: ping. Just wanted you to see the prior several lines about split-brain.
18:54 JoeJulian My guess is, that since you were not able to create split-brain, the self-heal engine fixed the stale file before you were able to down the second server.
18:54 Teela i think the problem im having is because i have the client and server on the same savers
18:54 Teela for node a & b
18:54 Teela so when i down the server
18:54 JoeJulian Which should only matter if you have a network partition.
18:55 Teela we do
18:55 Teela which is why stopping the server
18:55 Teela and editing the file doesnt seem to be creating a split brain
18:55 JoeJulian Two options. 1) Fix the network. :) or 2) Use quorum.
19:00 nshaikh joined #gluster
19:01 tdasilva joined #gluster
19:01 Teela i guess i could also use iptables?
19:01 Teela to block the traffic no?
19:05 JoeJulian To test creating a network partition? Sure.
19:08 Teela well the steps provided to create the split brains
19:08 Teela have all failed
19:08 Teela every time i do a info
19:08 Teela comes back clean
19:09 Teela so im thinking iptables might do the trick
19:09 Teela kinda of funny how its difficult to create in my dev
19:09 Teela but shows up in my prod just like crazy
19:09 Teela hahaha
19:10 JoeJulian Heh
19:11 JoeJulian dbruhn: bug 1184661 was closed, "wont fix"
19:11 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1184661 unspecified, unspecified, rc, steved, CLOSED WONTFIX, systemd service contains -w switch
19:29 B21956 So after we create thinly provisioned pools, bricks lvs and eventually a gluster volume, how does the volume expand?  The client only sees the 1GB and the Admin guide doesn't address the growth or hw to use mor of the pool.  Thought this was automatic...
19:30 calisto joined #gluster
19:31 Teela did ya resize the lvm after growing it?
19:31 Teela xfresize or something
19:31 Teela cant remember
19:32 JoeJulian Also, what's the page you're referring to? Perhaps that needs some clarification.
19:43 partner do we have any other channels i should be perhaps in for a reason or not ?
19:44 partner oh, topic :)
19:44 partner stupid me, always works to comment something to irc and immediately notice the answer :D
19:46 JoeJulian Asking the question engages parts of the brain that aren't always focused on the question.
19:46 gothos for some reason I always thought your where there as well since I'm around here
19:47 gothos awesome btw. that that marker bug got fixed as it appears, will have to backport to 3.6.2 I guess and confince my boss to let me install it
19:47 partner anyways, while i'm back for the moment a oneliner report from yesterdays meetup: good talks on ceph and gluster, it got even clearer there are use cases for both (and which ones) and RH has a fresh roadmap for both, niels gave good promo to get new people involved, pizza and beer was good also, karma++ for the meetup
19:47 glusterbot partner: karma's karma is now 0
19:47 partner thanks bot
19:49 partner keep the noise up, there's lots of interest out there
19:49 gothos the meetup in finnland?
19:49 partner yeah
19:49 gothos ah cool!
19:49 gothos btw will you be at fosdem?
19:50 partner i don't think so, i reviewed the program and i found pretty much nothing to my interest
19:51 partner maybe i looked the program wrong but its not targeted for me obviously
19:51 gothos ah okay :)
19:52 partner maybe in 2020 i can talk a few things about gluster somewhere if someone else is busy, now i'm still learning the basics :o
19:52 partner and i've decided to shut up until we reach the petabyte limit
19:53 partner "GlusterFS is a unified, poly-protocol, scale-out filesystem serving many petabytes of data. "
19:53 partner i count "many" as > 1 PB :)
19:53 partner only then i believe :)
19:54 gothos I'm only at 55TB, the drawback when you are working at a small institute at the university :/
19:55 partner gothos: but university, you can get money for research!11! :D
19:55 gothos but I don't doubt that it scales pretty well to PB
19:55 partner i'm only some 250 TB away from PB but need to arrange things to get that gap filled up
19:57 partner i wonder if there is any place to look into for a cheaper hardware that would be in general use here and there?
19:57 gothos yeah, sure. but I'm in IT only
19:57 gothos the stuff I hear from the researchers is rather painful
19:58 partner i'm mostly using somewhat "enterprise" approach ie. dell/hp hardware. nothing to complain about but its not exactly the cheapest ones to go along
19:58 gothos what are you currently using as raid chassis?
19:58 gothos we are using hp here as well with D2600, but they are quite bad when you consider the used space to hdd capacity ratio
19:58 samppah partner: hello! nice meeting you yesterday
19:59 samppah partner: have you looked at supermicro hardware?
19:59 partner samppah: hey, you made it alive :)
19:59 JoeJulian I really like the asus servers. We're all open compute here though.
19:59 partner samppah: remotely, its kind of known low-end producer, was wondering if there's something else, haven't done _any_ calculations towards anything at this point
20:00 partner JoeJulian: you've gone with the standard all around, racks and stuff (as is pretty much a must) ?
20:00 gothos I never knew asus was doing servers oO
20:00 gothos I guess anything with integrated multiple controllers might also work, but it sure ain't cheap
20:01 partner anyways, my setup is mainly dell boxes with 12x big disks and then hp boxes with 70x big disks
20:01 gothos what kind of hp boxes?
20:02 partner on top of that the usual cream ie. raid-6 arrays of 10-12 disks, lvm there, xfs there, bricks.. pretty basic stuff
20:02 gothos ditto withput lvm, too much pain
20:02 gothos even the xfs is annoying at the moment
20:02 partner gothos: good old mds600 at this point. just today looked into HP SL4500 series
20:02 gothos since I kind do an fscheck with only 32GB of memory
20:03 partner umm i lost you now gothos?
20:04 gothos partner: just saying that we are also using xfs here, but can't even do an xfs_check due to main memory being too small
20:05 JoeJulian Yes, all OCP.
20:05 gothos partner: those storage systems from hp look quite nice, guess I should talk to my boss
20:06 partner JoeJulian: nice, i recall on the not-so-new-anymore-job you had plenty of such stuff to work with, remotely remember seeing some links and stuff
20:07 JoeJulian https://plus.google.com/photos/113457525​690313682370/albums/6021614279363431345
20:07 partner gothos: there's plenty out there, we used to love sun x4500 boxes aka "thumper", 4 units, 48 disks, price wasn't bad
20:07 JoeJulian That module shown in the picture is all full now, of course.
20:09 partner nice, i wish i could get such stuff, need to take that into account as we've just moved (mostly) to a new datacenter and there's plenty of space still to build new rows of racks, just perhaps ocp in mind..
20:09 partner there are other parties interested on it aswell
20:09 partner internal that is
20:10 kmai007 joined #gluster
20:10 gothos partner: yeah, those are nice, I had one of them at another job at the university, but they are pretty pricey
20:10 partner gothos: well, i don't know much about the fsck, i've been running this setup now for 2 years and haven't had any issues with xfs for example
20:10 JoeJulian I like the hardware. The only issue we've had is the LSI cards but we're pretty sure we've identified a cold isle issue.
20:11 kmai007 guys i have a distri-rep.
20:11 kmai007 setup,
20:11 kmai007 i've deleted a directory in a volume
20:11 kmai007 through the client
20:11 kmai007 the directory is still listed, but i cannot stat it
20:11 kmai007 on the storage side i've tried to remove it, and it says no such dir. found
20:12 kmai007 how can i get rid of the empty dir. name?
20:12 JoeJulian On all the bricks/
20:12 JoeJulian ??
20:12 kmai007 all bricks
20:12 JoeJulian Wierd.
20:12 JoeJulian remount?
20:12 kmai007 on the client?
20:12 partner i'll go background to kill some zombies with my xbox now again before wife gets back.. :) bbl
20:12 JoeJulian Yeah
20:13 fubada joined #gluster
20:13 kmai007 tried that, no dice, even stop/start the volume no changes
20:13 kmai007 how do i go about deleting all in that volume without recreating the volume ?
20:13 JoeJulian Wait... the directory doesn't exist on any brick, but it keeps showing up on a client even after stopping and starting the volume. I can think of no way for that to be possible.
20:14 kmai007 the directory is listed ont he brick, but i cannot do anything to it
20:14 JoeJulian *can't
20:14 JoeJulian Oh!
20:15 JoeJulian so rm -rf $brick_root/$bad_directory fails with an error?
20:17 * JoeJulian should be more careful. Anyone copy-pasting that previous command without defining the variables... ouch.
20:19 gothos well, `set -u'
20:26 rwheeler joined #gluster
20:32 _Bryan_ joined #gluster
20:50 neofob left #gluster
21:03 MacWinner joined #gluster
21:09 vimal joined #gluster
21:14 booly-yam-4652 joined #gluster
21:51 ira joined #gluster
21:53 ira joined #gluster
21:56 yoavz Hi, after upgrading to the latest gluster I'm getting tons of this error: [2015-01-23 21:56:23.626705] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/ec100c65481c1e0fb39e03bd87b576a2.socket failed (Invalid argument)
21:56 yoavz any ideas?
22:00 JoeJulian My guess is that the services didn't get restarted.
22:01 JoeJulian stop glusterd, pkill -f gluster, start glusterd.
22:01 yoavz I tried to reboot the server and the problem stayed.
22:02 yoavz I'll to do this manually like you suggested, one minute
22:04 JoeJulian Well, reboot should have handled it.
22:05 yoavz I tried your suggestion but still, same error.
22:06 JoeJulian which version is this?
22:06 yoavz 3.6.2
22:08 yoavz The message "I [MSGID: 106006] [glusterd-handler.c:4257:__g​lusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 38 times between [2015-01-23 22:05:25.487695] and [2015-01-23 22:07:19.514419]
22:11 yoavz JoeJulian: it started after I tried to add geo-replication without the plugin installed.
22:11 yoavz I installed the plugin and updated gluster*
22:17 hchiramm_ joined #gluster
22:20 ProT-0-TypE joined #gluster
22:21 calisto joined #gluster
22:21 JoeJulian yoavz: is nfs disabled?
22:22 yoavz Not sure, let me check
22:22 yoavz nfs.disable: on
22:23 JoeJulian I bet that's why.
22:23 JoeJulian I thought they fixed that bug...
22:23 yoavz ^^<< more details needed :)
22:24 JoeJulian Those are informational messages. They have no effect on operation. My guess is that glusterd is trying to start the nfs service, no volumes have nfs enabled so nfs shuts down, glusterd tries to start it again...
22:25 yoavz Hmm, so it's safe? known bug?
22:26 JoeJulian It's safe. I thought I remembered seeing it before, but I don't know if it's still known. Probably would be a good idea for you to file a bug report.
22:26 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
22:27 yoavz I'll file the bug report... Thank you for your help! I really appreciate it.
22:27 JoeJulian You're welcome. :D
22:33 dgandhi joined #gluster
22:42 mator joined #gluster
22:59 PeterA joined #gluster
22:59 PeterA we having an issue with gluster NFS
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA lockd: server 10.101.165.68 not responding, timed out
22:59 PeterA NFS client lockd keep complaining
23:00 PeterA but we do not see related error on nfs.log on gluster node
23:00 JoeJulian /trout PeterA
23:00 doobi-sham-25434 joined #gluster
23:01 JoeJulian PeterA: Did you see the thread on gluster-devel about your quota issue?
23:01 PeterA yes
23:01 PeterA i replied
23:01 PeterA i tried on QA
23:01 PeterA worked
23:01 PeterA tried on prod, took down all the bricks on that volume
23:01 PeterA start force to bring it back and quota fix…
23:02 JoeJulian if course.. ugh...
23:02 JoeJulian Must not have replied to the list.
23:03 PeterA ?
23:04 T3 thanks JoeJulian and partner
23:04 JoeJulian You're welcome.
23:04 yoavz Now I have a different problem with geo-replication: [root@storage01 ~]# gluster volume geo-replication web-logs storage01.fr::web-logs-geo create push-pem force \\ Unable to fetch master volume details. Please check the master cluster and master volume. \\ geo-replication command failed
23:05 T3 I'm running kernel 3.13.0-44-generic with glusterfs 3.5.3
23:06 T3 and getting lots of stalls when reading or writing on gluster clients, with the error message I already pasted (will paste again):
23:07 T3 [2014-12-15 16:44:42.077836] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-site-images-client-1: remote operation failed: Transport endpoint is not connected. Path: /templates/apache/template/prod/facebook/10256093 (00000000-0000-0000-0000-000000000000)
23:07 T3 [2014-12-15 16:44:42.111496] I [socket.c:3027:socket_submit_request] 0-site-images-client-1: not connected (priv->connected = 0)
23:07 T3 [2014-12-15 16:44:42.111557] W [rpc-clnt.c:1488:rpc_clnt_submit] 0-site-images-client-1: failed to submit rpc-request (XID: 0x28493229x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (site-images-client-1)
23:07 JoeJulian PeterA: wrt lockd, I've never seen that. I would check to see what's actually listening for lockd and see if you're looking at the relevant log, I guess...
23:07 T3 any pointing on what to look at would be really helpful
23:08 JoeJulian T3: Looks like a network issue.
23:09 JoeJulian ~pasteinfo | yoavz
23:09 glusterbot yoavz: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
23:10 PeterA hmm i m on ubuntu and just noticed lockd is running
23:10 PeterA how do i make sure it's not run?
23:10 yoavz JoeJulian: https://dpaste.de/18V7
23:14 T3 JoeJulian, right, will try and get some network stats to check that
23:15 JoeJulian yoavz: Looks fine. You're going to have to look to see if a log gives any more clue.
23:16 JoeJulian PeterA: Switch to an EL build, or arch? ;) You'll have to find the upstart conf file and rename it with an extension that's not .conf.
23:16 PeterA .....
23:42 elico joined #gluster
23:53 hchiramm_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary