Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-02-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 cpetersen_ I brought brick 2 online.  As soon as quorum was back to normal, boom, the VM boots up.
00:01 haomaiwang joined #gluster
00:18 plarsen joined #gluster
00:24 ovaistariq joined #gluster
00:26 plarsen joined #gluster
00:33 cpetersen_ JoeJulian: I took brick2 down without taking the whole host down and the same problem occurs, quorum errors.
00:34 JoeJulian Or something that presents itself as quorum errors.
00:34 cpetersen_ I set quorum to fixed at 1 because realistically, I want to be able to lose 2 bricks and still survive.  I failed brick 2 and the VMs still boot up.
00:35 cpetersen_ This is not for a VM farm, it's for only 2 critical systems VMs which have DR backups done to them as well.
00:35 JoeJulian That's "no quorum at all".
00:35 cpetersen_ In the docs it says that auto quorum will make sure that half of the bricks are up.
00:35 cpetersen_ What is half of 3?
00:36 JoeJulian 2.
00:36 JoeJulian int(replica/2)+1
00:36 cpetersen_ Won't it just naturally always be out of quorum with auto then?  I should essentially set it to 2 right?
00:37 cpetersen_ But then I will never be able to stay up and running if two of them die, correct?
00:37 JoeJulian It will be auto by default if you have replica 3 or more.
00:37 JoeJulian correct
00:37 JoeJulian Or if it *thinks* that two have dies.
00:37 JoeJulian s/dies/died/
00:37 glusterbot What JoeJulian meant to say was: Or if it *thinks* that two have died.
00:38 cpetersen_ So in this case, the bug is that it *thinks* two have died, when in actuality, only 1 has died.
00:38 JoeJulian Seems to be, yes.
00:38 cpetersen_ That is the hypothesis here.  OK.  At least I have a succinct bug report now.  :)
00:38 cpetersen_ Effectively, if I set it to fixed at 2, I can survive 1 failure which is still fine.
00:39 JoeJulian But, like, your switch fails. All servers are isolated. You don't want them all deciding that the rest of the servers have died and booting up all the VMs.
00:39 cpetersen_ Then beyond that, I take the data from a surviving replica and rebuild the other bricks.
00:39 JoeJulian My scenario is why you don't want a quorum of 1.
00:39 cpetersen_ Right.
00:40 JoeJulian You could try a fixed 2.
00:40 JoeJulian Couldn't be worse.
00:41 cpetersen_ So if there is a quorum of 2 and brick3 fails, bricks1 and 2 are good to go.  If I lose brick1 due to network failure, how will brick2 behave?  Technically both of these guys think they are their own cluster now right?
00:42 cpetersen_ Sorry, lose brick1 immediately following that is what I meant.
00:42 JoeJulian If you only have one working server, you want it to fail.
00:42 JoeJulian Because it has no way of knowing that it's the only working server.
00:42 cpetersen_ So which one will impose a failure upon itself?  1 or 2?
00:43 cpetersen_ Is there a precidence that is set?
00:44 cpetersen_ Or both?
00:44 JoeJulian The volume quorum is handled by the client. If it has connections to two servers (in your case) it will be in quorum and will function. If it can only reach one server, it can only assume the one server it can reach is part of a network partition and it goes read-only.
00:45 cpetersen_ Ohhhh, IC.  I was thinking quorum was server-side.
00:45 JoeJulian The server quorum is handled by ping. As long as a server can ping one other server, it remains in quorum. If it loses quorum, it shuts down its bricks.
00:45 cpetersen_ Right, that's why those are two separate parameters in the Vcfg.
00:45 cpetersen_ Right, OK.
00:46 cpetersen_ I understand why 1 is very bad.
00:46 cpetersen_ =)
00:46 cpetersen_ 2 it is!
00:46 JoeJulian (without fencing)
00:46 JoeJulian :D
00:47 cpetersen_ Aha!  And as I set the quorum-count to 2, the bug presents itself.
00:47 cpetersen_ No writes allowed, jerk!
00:47 cpetersen_ lol
00:48 JoeJulian So your ganesha clients are *not* connecting to all the servers?
00:48 JoeJulian firewall?
00:48 JoeJulian hostname eval?
00:48 cpetersen_ Errr...
00:48 cpetersen_ When you say connecting to all of the servers, do you mean setting up NFS with multiple hosts?
00:49 JoeJulian ganesha uses libgfapi as a "client".
00:49 cpetersen_ Or just simply the NFS client on VMware not being able to reach all of the bricks?
00:49 JoeJulian @glossary
00:49 glusterbot JoeJulian: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
00:49 JoeJulian So ganesha *is* a gluster client.
00:49 cpetersen_ Right.
00:49 JoeJulian As such, it needs to be able to connect to all the bricks as shown in "gluster volume status"
00:51 cpetersen_ Well it says it can't reach 0-SHARED_vmvol01-client-2, but if I bring it back online it never has that error again indicating it can't reach any of the other bricks.
00:52 cpetersen_ The gluster v status is tickedy boo.
00:53 cpetersen_ I am taking off to go home, I can't stand sitting in this DC anymore today.
00:53 cpetersen_ =)
00:53 cpetersen_ Thanks for your help and I will touch base with you again tomorrow.  Really getting somewhere here!!!
00:53 JoeJulian Oh, good.
00:53 JoeJulian I really hope you figure it out so I don't have to.
00:53 cpetersen_ LOL
00:54 cpetersen_ I would make a money joke, but we're on freenode amongst all of this open source brain space, I won't go there.
00:54 JoeJulian hehe
00:54 cpetersen_ You're awesome, I thank you again for your help in all of this!
00:55 JoeJulian Careful... I already need an office with 10 foot ceilings so I can fit my ego.
01:01 haomaiwang joined #gluster
01:16 EinstCrazy joined #gluster
01:24 dthrvr joined #gluster
01:25 calavera joined #gluster
01:28 lanning bah, I have an open skylight...
01:28 JoeJulian :D
01:28 Lee1092 joined #gluster
01:33 farhorizon joined #gluster
01:38 theron joined #gluster
01:40 amye joined #gluster
01:42 EinstCrazy joined #gluster
01:59 harish joined #gluster
02:05 muneerse joined #gluster
02:09 muneerse2 joined #gluster
02:13 haomaiwa_ joined #gluster
02:14 theron joined #gluster
02:15 calavera joined #gluster
02:16 skoduri joined #gluster
02:21 muneerse joined #gluster
02:28 nishanth joined #gluster
02:38 haomaiwa_ joined #gluster
02:39 haomaiwang joined #gluster
02:40 haomaiwang joined #gluster
02:41 haomaiwang joined #gluster
02:45 haomaiwa_ joined #gluster
02:45 haomaiwang joined #gluster
02:46 16WAADEVP joined #gluster
02:46 haomaiwang joined #gluster
02:46 haomaiwang joined #gluster
02:47 haomaiwang joined #gluster
02:48 7YUAADLCJ joined #gluster
02:49 haomaiwa_ joined #gluster
02:50 haomaiwa_ joined #gluster
02:51 7YUAADLDG joined #gluster
02:52 haomaiwa_ joined #gluster
02:53 haomaiwa_ joined #gluster
02:54 haomaiwang joined #gluster
02:55 haomaiwa_ joined #gluster
02:56 haomaiwang joined #gluster
02:57 haomaiwa_ joined #gluster
02:58 haomaiwang joined #gluster
02:58 kenhui joined #gluster
02:59 haomaiwang joined #gluster
03:00 haomaiwang joined #gluster
03:01 haomaiwa_ joined #gluster
03:02 haomaiwang joined #gluster
03:03 haomaiwang joined #gluster
03:04 haomaiwa_ joined #gluster
03:05 haomaiwa_ joined #gluster
03:06 haomaiwang joined #gluster
03:07 arcolife joined #gluster
03:07 haomaiwang joined #gluster
03:08 haomaiwa_ joined #gluster
03:09 haomaiwa_ joined #gluster
03:09 arcolife joined #gluster
03:10 haomaiwang joined #gluster
03:11 haomaiwang joined #gluster
03:12 haomaiwa_ joined #gluster
03:13 haomaiwang joined #gluster
03:14 haomaiwa_ joined #gluster
03:14 nehar joined #gluster
03:15 haomaiwa_ joined #gluster
03:15 hackman joined #gluster
03:16 haomaiwang joined #gluster
03:17 64MAADV1R joined #gluster
03:18 haomaiwa_ joined #gluster
03:19 haomaiwang joined #gluster
03:20 haomaiwang joined #gluster
03:21 haomaiwang joined #gluster
03:22 haomaiwang joined #gluster
03:23 haomaiwang joined #gluster
03:24 haomaiwang joined #gluster
03:25 haomaiwa_ joined #gluster
03:26 haomaiwa_ joined #gluster
03:27 haomaiwang joined #gluster
03:27 sacrelege joined #gluster
03:28 haomaiwa_ joined #gluster
03:28 overclk joined #gluster
03:29 haomaiwa_ joined #gluster
03:30 calavera joined #gluster
03:30 haomaiwa_ joined #gluster
03:31 haomaiwa_ joined #gluster
03:32 haomaiwang joined #gluster
03:32 rcampbel3 joined #gluster
03:33 haomaiwang joined #gluster
03:34 haomaiwang joined #gluster
03:36 92AAACXGP joined #gluster
03:37 haomaiwa_ joined #gluster
03:38 haomaiwang joined #gluster
03:39 haomaiwa_ joined #gluster
03:40 haomaiwang joined #gluster
03:41 haomaiwa_ joined #gluster
03:42 7GHAAB0XV joined #gluster
03:43 haomaiwang joined #gluster
03:44 atinm joined #gluster
03:44 haomaiwa_ joined #gluster
03:45 haomaiwa_ joined #gluster
03:46 F2Knight joined #gluster
03:46 haomaiwa_ joined #gluster
03:47 haomaiwa_ joined #gluster
03:48 haomaiwang joined #gluster
03:49 haomaiwa_ joined #gluster
03:50 haomaiwa_ joined #gluster
03:51 haomaiwa_ joined #gluster
03:52 haomaiwang joined #gluster
03:53 haomaiwa_ joined #gluster
03:53 itisravi joined #gluster
03:54 haomaiwang joined #gluster
03:55 haomaiwang joined #gluster
03:56 haomaiwang joined #gluster
03:57 haomaiwa_ joined #gluster
03:58 haomaiwa_ joined #gluster
03:59 haomaiwang joined #gluster
04:00 haomaiwa_ joined #gluster
04:01 haomaiwang joined #gluster
04:02 haomaiwang joined #gluster
04:03 kanagaraj joined #gluster
04:03 haomaiwang joined #gluster
04:04 haomaiwa_ joined #gluster
04:05 rcampbel3 joined #gluster
04:05 64MAADWJE joined #gluster
04:06 64MAADWJZ joined #gluster
04:07 64MAADWKL joined #gluster
04:08 nangthang joined #gluster
04:08 64MAADWK9 joined #gluster
04:09 64MAADWLO joined #gluster
04:10 haomaiwang joined #gluster
04:11 haomaiwang joined #gluster
04:12 haomaiwang joined #gluster
04:12 shubhendu joined #gluster
04:13 haomaiwang joined #gluster
04:13 ppai joined #gluster
04:14 nbalacha joined #gluster
04:14 haomaiwa_ joined #gluster
04:15 haomaiwa_ joined #gluster
04:15 nhayashi joined #gluster
04:16 haomaiwang joined #gluster
04:17 haomaiwang joined #gluster
04:18 92AAACXRV joined #gluster
04:19 kotreshhr joined #gluster
04:19 ppai_ joined #gluster
04:19 92AAACXR5 joined #gluster
04:20 haomaiwa_ joined #gluster
04:25 aravindavk joined #gluster
04:27 kenhui joined #gluster
04:28 deepakcs joined #gluster
04:40 hgowtham joined #gluster
04:47 calavera joined #gluster
04:47 karthikfff joined #gluster
04:48 RameshN joined #gluster
04:50 pppp joined #gluster
04:56 vmallika joined #gluster
04:56 ndarshan joined #gluster
04:56 jiffin joined #gluster
04:57 farhorizon joined #gluster
04:58 social joined #gluster
05:03 Bhaskarakiran joined #gluster
05:03 nehar joined #gluster
05:13 jeek joined #gluster
05:18 rafi joined #gluster
05:24 skoduri joined #gluster
05:26 gowtham joined #gluster
05:29 hackman joined #gluster
05:36 kshlm joined #gluster
05:36 ashiq joined #gluster
05:39 python_lover joined #gluster
05:47 calavera joined #gluster
05:48 poornimag joined #gluster
05:48 sas joined #gluster
05:52 Ulrar joined #gluster
05:53 kdhananjay joined #gluster
05:54 Manikandan joined #gluster
05:56 nishanth joined #gluster
05:59 ramky joined #gluster
05:59 Saravanakmr joined #gluster
06:05 morse joined #gluster
06:14 shruti joined #gluster
06:15 anil joined #gluster
06:15 sakshi joined #gluster
06:21 ovaistariq joined #gluster
06:22 ovaistar_ joined #gluster
06:29 anil joined #gluster
06:30 nangthang joined #gluster
06:31 rcampbel3 joined #gluster
06:33 atalur joined #gluster
06:37 Bhaskarakiran joined #gluster
06:42 gowtham joined #gluster
06:45 gem joined #gluster
06:46 Wizek_ joined #gluster
06:57 Bhaskarakiran_ joined #gluster
06:58 sathees joined #gluster
07:02 hchiramm joined #gluster
07:06 auzty joined #gluster
07:10 nehar joined #gluster
07:15 ira joined #gluster
07:19 Wizek joined #gluster
07:22 jtux joined #gluster
07:23 itisravi joined #gluster
07:27 robb_nl joined #gluster
07:28 rcampbel3 joined #gluster
07:32 jtux joined #gluster
07:33 itisravi joined #gluster
07:33 itisravi joined #gluster
07:34 itisravi joined #gluster
07:55 [diablo] joined #gluster
08:06 jri joined #gluster
08:11 [Enrico] joined #gluster
08:15 nehar joined #gluster
08:21 jwd joined #gluster
08:35 harish_ joined #gluster
08:36 deniszh joined #gluster
08:38 ivan_rossi joined #gluster
08:42 Akee joined #gluster
08:47 fsimonce joined #gluster
08:55 aravindavk joined #gluster
08:57 nehar joined #gluster
08:57 haomaiwa_ joined #gluster
09:01 haomaiwang joined #gluster
09:02 haomaiwa_ joined #gluster
09:08 Slashman joined #gluster
09:19 kshlm joined #gluster
09:23 hackman joined #gluster
09:25 atinm joined #gluster
09:26 karnan joined #gluster
09:27 robb_nl joined #gluster
09:27 nehar joined #gluster
09:28 [diablo] joined #gluster
09:38 The_Ball joined #gluster
09:38 The_Ball Anyone seen this: [2016-02-23 08:58:34.755270] E [client-rpc-fops.c:5061:client3_3_getxattr] (-->/lib64/libglusterfs.so.0(syncop_getxattr+0x1a0) [0x7f659d954840] -->/usr/lib64/glusterfs/3.7.8/xlator/protocol/client.so(client_getxattr+0x75) [0x7f659003afc5] -->/usr/lib64/glusterfs/3.7.8/xlator/protocol/client.so(client3_3_getxattr+0x584) [0x7f6590051184] ) 0-: Assertion failed: 0
09:38 glusterbot The_Ball: ('s karma is now -123
09:42 kdhananjay joined #gluster
09:43 gem joined #gluster
09:56 gem_ joined #gluster
10:01 haomaiwang joined #gluster
10:08 [Enrico] joined #gluster
10:10 hchiramm joined #gluster
10:14 gowtham joined #gluster
10:22 Wizek_ joined #gluster
10:26 kdhananjay joined #gluster
10:28 refj joined #gluster
10:30 refj I need to upgrade a minor version of glusterfs. Should I unmount the clients and stop the glusterfs daemons before proceeding? I have found documentation describing upgrades between major version, but not any notes concerning minor versions.
10:51 anil_ joined #gluster
10:57 sacrelege_ joined #gluster
11:00 nehar joined #gluster
11:01 haomaiwa_ joined #gluster
11:02 s-hell Can anyone help me with this error: https://paste.pcspinnt.de/view/19bdb514
11:02 glusterbot Title: Untitled - Paster pcpsinnt.de (at paste.pcspinnt.de)
11:03 DV joined #gluster
11:03 Manikandan REMINDER: Gluster Community Bug Triage meeting in #gluster-meeting at 12:00 UTC (~in 60 minutes)
11:27 kdhananjay joined #gluster
11:32 refj I went ahead a patchet without stopping gluster. everything went fine.
11:32 gowtham joined #gluster
11:33 ovaistariq joined #gluster
11:35 kkeithley joined #gluster
11:37 nehar_ joined #gluster
11:38 jwd joined #gluster
11:43 robb_nl joined #gluster
11:52 EinstCrazy joined #gluster
11:58 kshlm joined #gluster
12:00 hgowtham_ joined #gluster
12:01 haomaiwa_ joined #gluster
12:09 Wizek joined #gluster
12:11 Ulrar joined #gluster
12:12 Bhaskarakiran_ joined #gluster
12:16 voobscout joined #gluster
12:16 vmallika joined #gluster
12:17 pppp joined #gluster
12:17 hchiramm joined #gluster
12:21 python_lover joined #gluster
12:25 Manikandan joined #gluster
12:36 voobscout joined #gluster
12:45 nehar joined #gluster
12:52 chirino joined #gluster
12:53 atinm joined #gluster
12:58 robb_nl joined #gluster
12:59 haomaiwa_ joined #gluster
13:03 Bhaskarakiran_ joined #gluster
13:03 johnmilton joined #gluster
13:07 voobscout joined #gluster
13:13 anil_ joined #gluster
13:15 unclemarc joined #gluster
13:18 nehar joined #gluster
13:19 pdrakeweb joined #gluster
13:20 kdhananjay joined #gluster
13:21 hgowtham_ joined #gluster
13:28 haomaiwa_ joined #gluster
13:30 skoduri joined #gluster
13:33 voobscout joined #gluster
13:42 vmallika joined #gluster
13:47 wolsen joined #gluster
13:49 voobscout joined #gluster
13:55 sebamontini joined #gluster
13:55 sebamontini hello
13:55 glusterbot sebamontini: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
13:56 sebamontini lol, sorry, here's my question: I have 2 servers with gluster 3.7.2, in distributed replica, with 2 disk each (one of 50gb and one of 300gb, each node)
13:56 sebamontini i need to decomission the 50gb disks in both nodes
13:57 sebamontini right now the data is distributed in both disks (rebalanced and everything), how can i remove the 50gb brick from both nodes without loosing any data?
13:57 julim joined #gluster
13:58 ahino joined #gluster
13:58 sebamontini here's a pastebin of what i've got (that i explained before) http://paste.nubity.com/ab910c216ce0d214.avrasm
13:58 glusterbot Title: Paste | Nubity (at paste.nubity.com)
13:59 dariol joined #gluster
14:00 post-factum sebamontini: https://gluster.readthedocs.org/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes
14:01 sebamontini post-factum yes, that is what i was reading 5 min ago
14:01 post-factum sebamontini: then, well, you've got that! :)
14:01 sebamontini but i was concern about the note "Data residing on the brick that you are removing will no longer be accessible at the Gluster mount point. "
14:01 haomaiwa_ joined #gluster
14:02 sebamontini and in the last point (item 5) it says "Rebalance the volume to ensure that all files are distributed to the new brick.
14:02 sebamontini You can use the rebalance command as described in Rebalancing Volumes"
14:03 sebamontini how can i do a rebalance of just one brick? if the old 50gb brick is no logen in the volume, what happens to that data?
14:03 post-factum yup, doc is pretty scary about that
14:03 arcolife joined #gluster
14:03 sebamontini i mean...what i need to do is...unbalance what i have now, move all the data to the brick2, then remove the brick1
14:03 post-factum i guess, on remove-brick start you definitely start rebalancing data to remaining bricks. so, after it finishes, but before commit, just check, that data are on remaining brick
14:04 kenhui joined #gluster
14:04 kenhui joined #gluster
14:04 sebamontini ok, so when i remove the brick, the data on the old brick will copy itself to the remaingin bricks? no data loss?
14:05 sebamontini man that readthedocs was scary! LOL
14:08 voobscout joined #gluster
14:08 post-factum i'm pretty sure gluster shouldn't make your data disappear
14:10 skylar joined #gluster
14:12 sebamontini awesome
14:12 sebamontini thanks a lot post-factum
14:14 post-factum np
14:17 Ulrar So I have files on "Possibly undergoing heal", what does that mean ?
14:17 voobscout joined #gluster
14:17 Ulrar Is the file usable ?
14:20 atinm joined #gluster
14:21 mhulsman joined #gluster
14:25 theron joined #gluster
14:25 post-factum Ulrar: have you performed some management actions on volume?
14:26 Ulrar No
14:26 Ulrar Well, proxmox might have, no idea what it does, but I don't think so
14:34 post-factum Ulrar: and the files under "Possibly undergoing heal" stay the same and do not disappear?
14:34 julim_ joined #gluster
14:34 Ulrar post-factum: When I shutdown the VM, the file stays there but the "Possibly undergoing heal" disappear
14:34 Ulrar But appart from that, yes, they stay the same
14:35 Ulrar Managed to restart the important VM after all, looks more like a proxmox bug than a problem with glusterfs. The heal part still seems weird though
14:35 Ulrar Ha wait, one of the files just disappeared
14:37 jiffin joined #gluster
14:41 Ulrar post-factum: Do you know if a file is supposed to be usable during a heal ? Seems like the VM "freez" at boot after a problem, as if it was just waiting to access it's disk
14:44 kotreshhr left #gluster
14:46 voobscout joined #gluster
14:46 post-factum Ulrar: AFAIK, healing is blocking op wight now
14:46 post-factum Ulrar: but we should cast some dev here
14:47 post-factum JoeJulian: ^^
14:47 post-factum *right now
14:47 hamiller joined #gluster
14:47 shaunm joined #gluster
14:47 post-factum I indeed used to observe VM freezes while healing
14:48 post-factum that is why I do not store VM images on GlusterFS :)
14:48 Ulrar What do you use ?
14:48 Ulrar Ceph ?
14:48 post-factum yes
14:48 Ulrar Guess I'l switch then
14:48 Ulrar Never used Ceph, will it work fine on 3 nodes ?
14:48 post-factum but I use GlusterFS for everything else -- for file storage
14:49 post-factum it will. 3 nodes is a recommended minimum
14:49 Ulrar Well for regular file I find it very slow, but I sometimes use it
14:49 Ulrar Migrating production servers will be great ..
14:50 post-factum CephFS is a headache now, so there is no real alternative for GlusterFS
14:50 Ulrar Well blocking the files during healing is just not acceptable, the HA just doesn't work
14:50 post-factum you can try sharding for volume that holds huge images
14:51 Ulrar Yeah but it'll still need to block the shard, and thus the whole VM, no ?
14:52 post-factum shard is smaller than whole image, so it should be processed faster
14:52 Ulrar Guess it's worth a try, but another downtime like that and I'm going to have a lot of trouble :)
14:53 post-factum also, I saw some patch related to background heal
14:53 Ulrar If only the network was reliable
14:53 Ulrar post-factum: Ha ? For 3.7 or 3.8 ?
14:53 kdhananjay joined #gluster
14:54 post-factum let me find it first
14:55 post-factum otherwise my assumption could be just wrong
14:56 post-factum hmm http://review.gluster.org/#/c/13207/
14:56 glusterbot Title: Gerrit Code Review (at review.gluster.org)
14:56 post-factum but that is client-side heal
14:56 post-factum I definitely need dev mentoring to figure it out
14:58 Ulrar Might just disable HA for now .. It's better to wait for a human 10 minutes to start the VM again on another server if needed than having to wait forever for the heal to finish
14:58 Ulrar At least until I have a reliable solution
14:58 post-factum try ceph rbd instead for vm images
14:59 post-factum and glusterfs for usual files
14:59 farblue joined #gluster
15:00 Ulrar Well, the problem will be to install ceph, which I never used, on the servers already filled up by glusterfs
15:00 Ulrar Can I shrink a glusterfs volume ?
15:00 honzik666 post-factum: do you run the monitoring nodes on separate machines from the storage nodes? I am also considering switching to ceph for block storage. Would 3 physical machines be sufficiently robust for the setup? I am assuming each machine would run a monitoring node as well as the storage node
15:01 Ulrar Could create the two volumes and plan a downtime some night to switch the disks to ceph
15:01 post-factum Ulrar: what is your disk layout?
15:01 Ulrar post-factum: Have three nodes, with 1,1 Tb on each, with a replica of 3
15:01 haomaiwa_ joined #gluster
15:01 post-factum honzik666: running mon and osd on same node is ok
15:01 Ulrar 3 might be too much in hindisght, but that's what I have now :)
15:02 post-factum Ulrar: well, i mean, can you shrink brick partitions?
15:02 post-factum if you have XFS underneath, you cannot
15:02 Ulrar Ha, it's XFS
15:02 post-factum :D
15:02 Ulrar Great
15:02 honzik666 post-factum: thank you for the hint, I couldn't decide on the final setup
15:02 amye joined #gluster
15:03 Ulrar Well, since I have a replica of 3, I could probably kill two nodes without any problems
15:03 Ulrar And use a VM as the third node for the time of the migration
15:03 ctria joined #gluster
15:03 Ulrar Then remove gluster from the last node, replace it by ceph, and remove the VM
15:03 post-factum sounds like a plan
15:04 Ulrar I'll look into that then ! thanks a lot
15:04 post-factum np
15:05 coredump|br joined #gluster
15:05 post-factum honzik666: generally, for large production it is better to have separate osds and mons, but i guess if the cluster is 3 nodes big, it is ok to combine
15:05 post-factum we combine, and it works
15:06 post-factum ceph mons do not require lots of resources, just availability
15:06 honzik666 post-factum: ok, what bandwidth do you have among the nodes if it's not secret? I am assuming you the storage on separate VLAN
15:06 post-factum 2×10G
15:06 wnlx_ joined #gluster
15:06 post-factum bonding
15:06 post-factum interconnected with some junipers
15:07 kdhananjay Ulrar: just saw that you are running VMs on gluster. You might want to try sharding and arbiter for your use-case, in case you are not using it already.
15:07 honzik666 post-factum: that sounds sufficient ;-)
15:07 ahino joined #gluster
15:07 farblue Hi all :) Is it possible to grow an existing dispersed volume by just adding bricks?
15:08 post-factum honzik666: a bit of overkill, but i insisted on bonding in the name of reliability, and 1G is not sufficient
15:08 post-factum so, 2×10
15:08 post-factum kdhananjay: what about heals? is that blocking op?
15:08 post-factum farblue: afaik, no
15:09 kdhananjay post-factum: heals would heal only the modified shard.
15:09 honzik666 post-factum: I am wondering if 2x1G would be sufficient for some moderately loaded environment
15:09 robb_nl joined #gluster
15:10 kdhananjay post-factum: if we assume you choose shard block size to be 128M, then self-heal will only heal files of that size as opposed to trying to heal the whole X {G,T}B image.
15:11 farblue post-factum: thanks :)
15:11 kdhananjay post-factum: https://www.gluster.org/pipermail/gluster-users/2015-December/024568.html
15:11 glusterbot Title: [Gluster-users] Sharding - what next? (at www.gluster.org)
15:12 post-factum kdhananjay: but healing still blocks the piece being healed?
15:12 kdhananjay post-factum: that was feedback we got from a community user named Lindsay who tested it extensively.
15:12 farblue left #gluster
15:12 post-factum Ulrar: take a look ^^
15:13 post-factum kdhananjay: thx for the info
15:13 kdhananjay post-factum: only until the much smaller 128M block is healed. No biggie in comparison to the terabytes of data that otherwise need to be compared between the good and the bad copy to figure out where the heal is needed.
15:14 post-factum kdhananjay: ok, so I got it right
15:14 kdhananjay post-factum: in our testing, we'd found heal to take a few minutes, as opposed to ...well sometimes days for vms with size of the order of few TBs.
15:14 plarsen joined #gluster
15:15 kdhananjay post-factum: right.
15:16 post-factum honzik666: 2×1 is a minimum recommended, so it will work for moderate load. but pay an attention to latency as well, 10G should make it smaller
15:16 post-factum smaller litency -- higer iops
15:16 post-factum s/litency/latency/
15:16 glusterbot What post-factum meant to say was: smaller latency -- higer iops
15:17 voobscout joined #gluster
15:18 honzik666 post-factum: that makes perfect sense, thank you
15:26 farhorizon joined #gluster
15:27 harish joined #gluster
15:28 ravage joined #gluster
15:28 kenhui joined #gluster
15:28 kenhui joined #gluster
15:30 Ulrar kdhananjay: Interesting, I am not using shards atm
15:30 Ulrar Is it possible to add that to an existing volume ?
15:31 kdhananjay Ulrar: yes you can enable sharding on an existing volume but the already existing files will continue to remain unsharded. new files that are created will definitely be sharded.
15:31 nbalacha joined #gluster
15:31 Ulrar kdhananjay: Ha, I see
15:31 Ulrar Any way of "correcting" that ? Maybe moving the file out and in again ?
15:32 Ulrar Would imply a downtime though
15:32 kdhananjay Ulrar: if you want to move your existing files to sharded mode, you can create a new volume, enable shard on it, and move your existing images from the old volume to the new volume.
15:32 voobscout joined #gluster
15:33 kdhananjay move/copy whatever works
15:35 Ulrar kdhananjay: Moving the file out of the volume and in again won't work ? Creating a new volume, moving all the files and changing the proxmox conf seems harder
15:35 ovaistariq joined #gluster
15:36 kdhananjay Ulrar: thinking ..
15:39 kanagaraj joined #gluster
15:40 kdhananjay Ulrar: move out all the data to local fs, enable sharding on this volume, and then move them back to the volume, maybe?
15:40 kdhananjay Ulrar: would you be okay with that?
15:43 Ulrar Enabling sharding isn't long I guess ?
15:45 kdhananjay Ulrar: didn't get you.
15:46 Ulrar kdhananjay: I mean enabling sharding doesn't take hours, it's just a setting ?
15:46 Ulrar I'm looking that up now
15:46 sacrelege joined #gluster
15:46 sacrelege joined #gluster
15:46 kdhananjay Ulrar: it will be over before you know it. ;)
15:47 Ulrar Perfect !
15:47 post-factum Ulrar, kdhananjay: I've tested enabling sharding on existing volume, and copying file within that volume to temporary file and renaming it back also works
15:48 Ulrar Ha, great
15:48 Ulrar That way I can do it VM by VM
15:48 Ulrar Easier to plan
15:49 Ulrar So I just have to do "gluster volume set dis-rep features.shard on" and set the sharding size, is that it ?
15:49 kdhananjay post-factum: thinking ..
15:49 Ulrar And the different parameters mentionned in your link, since it's for VMs
15:49 kdhananjay Ulrar: those parameters can be enabled using 'gluster volume set <VOL> group virt' command
15:50 kdhananjay Ulrar: yes you are right about "gluster volume set dis-rep features.shard on"
15:50 kdhananjay Ulrar: and yes, you can choose the shard size you prefer. default value is 4M
15:50 post-factum ppl told me here they've tested shards up to 512M
15:51 Ulrar I'm guessing the bigger the better for performances but the worse for healing ?
15:51 post-factum worse for healing, but not necessarily better for performance
15:51 post-factum saying, you have distributed volume, then shards will be distributed as well
15:52 kdhananjay Ulrar: yeah, 512M was found to be providing a nice balance between the two, in our testing at least.
15:54 sacrelege joined #gluster
15:54 post-factum as i understand, shards are stored within same folder on brick, and i would like not to recommend storing more than several hundred of files within same folder. so saying you have 8G VM image, shard size of ^$M sounds reasonable for me
15:54 post-factum oops
15:54 post-factum *shard size of 64M
15:55 Ulrar I see
15:55 Ulrar Most of the VMs currently have 25 gb disks, and between 50 and 200 gb of disk as secondary
15:56 jwaibel joined #gluster
15:56 post-factum well, 256M should work
15:56 post-factum at least, you see the acceptable range
15:57 post-factum and reasonable values
15:57 Ulrar Yeah, sure. Thanks !
15:57 post-factum kdhananjay++
15:57 glusterbot post-factum: kdhananjay's karma is now 5
15:57 Ulrar kdhananjay++
15:57 glusterbot Ulrar: kdhananjay's karma is now 6
15:57 Ulrar post-factum++
15:57 glusterbot Ulrar: post-factum's karma is now 1
15:58 kdhananjay Ulrar++ post-factum++
15:58 Ulrar Great
15:58 glusterbot kdhananjay: Ulrar's karma is now 1
15:58 glusterbot kdhananjay: post-factum's karma is now 2
15:58 post-factum ok, Ulrar++ :)
15:58 glusterbot post-factum: Ulrar's karma is now 2
16:00 kdhananjay post-factum: your idea will work. :)
16:00 kdhananjay Ulrar: ^^
16:00 neofob joined #gluster
16:01 haomaiwa_ joined #gluster
16:02 julim joined #gluster
16:02 skoduri joined #gluster
16:04 sebamontini hey guys, me again...i'm creating a dev-test scenario to play around with glusterFS in AWS
16:04 sebamontini pretty simple, just 2 ec2 instances, 1 brick each, in replica 2
16:04 sebamontini http://paste.nubity.com/4c230b9219f5a9da.vhdl
16:04 glusterbot Title: Paste | Nubity (at paste.nubity.com)
16:05 F2Knight joined #gluster
16:05 sebamontini but, on the second node (gluster02) i'm not seeing any file being replicated
16:05 sebamontini http://paste.nubity.com/d95ead4739568873.vhdl
16:05 glusterbot Title: Paste | Nubity (at paste.nubity.com)
16:06 sebamontini as you can see, there are 850mb of files created in vol01 in gluster01, that are not being replicated to gluster02
16:06 Ulrar Looks like the virt group doesn't exist
16:06 sebamontini any ideas what am i doing wrong?
16:06 Ulrar The mail you linked does mention it
16:08 kdhananjay Ulrar: you use debian too?
16:08 Ulrar Yeah
16:09 hagarth @replication
16:09 glusterbot hagarth: I do not know about 'replication', but I do know about these similar topics: 'check replication', 'geo-replication'
16:09 hagarth @check replication
16:09 glusterbot hagarth: http://joejulian.name/blog/quick-and-dirty-python-script-to-check-the-dirty-status-of-files-in-a-glusterfs-brick/
16:10 hagarth @client
16:10 kdhananjay Ulrar: yeah, then you will need to manually set these options individually :-/
16:10 glusterbot hagarth: I do not know about 'client', but I do know about these similar topics: 'client io-threads', 'files edited with vim become unreadable on other clients with debian 5.0.6'
16:10 RayTrace_ joined #gluster
16:10 sebamontini hagarth how do i check the replication?
16:10 sebamontini when i do volume info i se that it's a replica 1 x 2 = 2
16:11 calavera joined #gluster
16:11 JoeJulian "gluster volume heal $vol info"
16:11 JoeJulian In a perfect world, if there's nothing there that needs healing, the replicas are healthy.
16:11 Ulrar kdhananjay: Well, not that bad
16:12 sebamontini ok, let me check tat JoeJulian hagarth , thx
16:12 Ulrar kdhananjay: just to be sure, "gluster volume set vm-storage quick-read off" right ?
16:12 JoeJulian If you're concerned that the replicas may not be healthy (you manually deleted a bunch of stuff off a brick), you can run a full heal against all of your files with "gluster volume heal $vol full"
16:12 hagarth sebamontini: also make sure that you are not writing directly to the bricks
16:12 kdhananjay Ulrar: perfect
16:13 Ulrar kdhananjay: Thanks a lot for your help ! Have to go now, but I'll do all of that tomorrow
16:14 kdhananjay Ulrar: np!
16:14 sebamontini ok, so i found this: http://paste.nubity.com/b5a399ea2e392279.vhdl
16:14 glusterbot Title: Paste | Nubity (at paste.nubity.com)
16:14 JoeJulian glusterfsd should use inotify to watch the brick and die a horrible death if someone modifies the brick.
16:15 sebamontini hagarth i did not write to the brick, just to the vol
16:15 jwd joined #gluster
16:15 sebamontini maybe there's some networking issue, or some port closed or something
16:15 sebamontini ?
16:16 JoeJulian @ports
16:16 glusterbot JoeJulian: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up since 3.4.0 (24009 & up previously). (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
16:18 JoeJulian @forget ports
16:18 glusterbot JoeJulian: The operation succeeded.
16:19 sebamontini i'm using tcp as transport, and i have 38465-38467 and 24007-24008 open between instances
16:21 JoeJulian @learn ports as glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
16:21 glusterbot JoeJulian: The operation succeeded.
16:22 JoeJulian sebamontini: So nothing can communicate with the bricks, eh? That's not going to be very useful.
16:22 sebamontini i though glusterfsd comunicated locally with their bricks
16:23 sebamontini i cannot open 49152..."and up" in the security group :(
16:23 JoeJulian Does your volume reside on only one server?
16:23 sebamontini the volume is supposed to be replicated between the two ec2 instances
16:24 ninkotech joined #gluster
16:24 ninkotech_ joined #gluster
16:24 JoeJulian Assuming you know that you're only going to have one brick on each server, you can predict with some certainty that the brick port will always be 49152. If you're not so sure, open up a small range.
16:25 JoeJulian "gluster volume status" will show you which ports the bricks are using.
16:25 sebamontini but why when i did the peer probe i didn't show this error?
16:25 JoeJulian With some clever scripting, you could even use that to dynamically manage the security group using the CLI.
16:26 sebamontini ok, that sounds great
16:26 JoeJulian Because peer probing is used to create a management group between management daemons over the management port.
16:26 sebamontini it's replicating now!! :)
16:26 sebamontini thanks so much for your help....once again
16:26 JoeJulian You're welcome.
16:27 cpetersen_ JoeJulian:  Question.  Continuing on from yesterday, how do I check the gluster client to see what bricks it is actively connected to?
16:27 cpetersen_ Sorry, I'll try not to inflate your ego today... ;)
16:27 JoeJulian I usually just use netstat and compare with volume status.
16:29 wushudoin joined #gluster
16:30 cpetersen_ Also, when all nodes are up, what does it mean when the heal info command shows files but no other information?
16:30 JoeJulian It should mean those files are queued for heal.
16:31 cpetersen_ Shouldn't they just heal then?
16:31 cpetersen_ Automatically?
16:32 JoeJulian yes
16:32 cpetersen_ I think they have been sitting there like that for a while.
16:32 cpetersen_ I haven't touched this array since last night.
16:32 shubhendu joined #gluster
16:32 cpetersen_ I shouldn't have to kick off a manual heal in this case.
16:33 JoeJulian No, you shouldn't. There's a scheduler that should handle that. Check heal..statistics
16:33 RameshN joined #gluster
16:33 JoeJulian And, of course, glustershd.log.
16:34 cpetersen_ There are a bunch of these in that log.
16:34 cpetersen_ [2016-02-23 16:22:48.073987] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-SHARED_vmvol01-client-2: remote operation failed. Path: <gfid:7733da52-f58e-452f-8f82-0eeb8076262c> (7733da52-f58e-452f-8f82-0eeb8076262c) [No such file or directory]
16:34 cpetersen_ For both of the healthy bricks.
16:35 cpetersen_ And some of these as well.
16:35 cpetersen_ [2016-02-23 16:22:48.037143] W [MSGID: 114031] [client-rpc-fops.c:1512:client3_3_ftruncate_cbk] 0-SHARED_vmvol01-client-2: remote operation failed [Operation not permitted]
16:37 JoeJulian The second one is probably because you're not allowed to set xattrs on a file that doesn't exist.
16:37 cpetersen_ Right, because the node is down.
16:38 cpetersen_ Or not necessarily, rereading your statement, yes that makes snese.
16:39 JoeJulian client-2 is the third brick. Assuming that brick is up, the "no such file" warning *probably* means the file that had that gfid has been deleted.
16:39 cpetersen_ So do I ignore that message?
16:41 ekuric joined #gluster
16:41 JoeJulian I assume you know how to check the gfid path for that entry.
16:42 cpetersen_ I do, there are just a shit ton of them to check.
16:42 cpetersen_ Well, of the first error.  Less of the second.
16:44 JoeJulian Huh, I wonder why. Pick one. If that gfid doesn't exist on any server my assumption would be a stale entry in .glusterfs/indices/xattrop
16:44 cpetersen_ Will that be regular maintenance that I need to perform?
16:45 JoeJulian No[1].
16:45 JoeJulian [1] If it is, however, try to figure out where they're coming from and file a bug report.
16:45 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
16:48 cpetersen_ I picked one.  The gfid file exists on nodes 1 and 2, but not 3.
16:48 JoeJulian Ah, then I would leave it.
16:51 kshlm joined #gluster
16:51 cpetersen_ GIF resolver shows it to be the flat file of my VM.
16:51 cpetersen_ 029dfee6-23ad-448e-bb79-7f4661ea990d    ==      File: /mnt/sdb1/brick/DIR01/DIR01-flat.vmdk
16:52 cpetersen_ Pretty sure that file is on that brick... just perhaps another GFID...
16:53 voobscout joined #gluster
16:53 JoeJulian You can check the gfid of the file by looking at the ,,(extended attributes)
16:53 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://pl.atyp.us/hekafs.org/index.php/2011/04/glusterfs-extended-attributes/
16:55 cpetersen_ The GFID is the same across all nodes.
16:56 cpetersen_ On node 3, the 9d folder doesn't exist within <brick>/.glsuterfs/02
16:56 cpetersen_ :S
16:56 ivan_rossi left #gluster
16:57 JoeJulian So you have an xattrop entry, actual file, but no gfid hardlink. Interesting.
16:57 cpetersen_ Well there is no xattrop entry either.
16:58 cpetersen_ There is no "029dfee6-23ad-448e-bb79-7f4661ea990d" in the <brick>/.glusterfs/indices/xattrop/ folder.
16:58 cpetersen_ Wait, yes there is.  Sorry, I'm dumb.
16:58 JoeJulian Whew
16:59 cpetersen_ Autocomplete wasn't showing me.
16:59 * cpetersen_ tab tab tab tab
16:59 cpetersen_ I had to use my eyes.
16:59 cpetersen_ Assumptions were made...
16:59 JoeJulian Yeah, been there, done that.
17:00 cpetersen_ So if I understand correctly, because the client can't access that GFID on brick3, it's not in quorum if node 2 goes down?
17:00 cpetersen_ Or is that even related?
17:01 haomaiwa_ joined #gluster
17:01 cpetersen_ I feel like I've done something wrong.  :(
17:02 JoeJulian Should be unrelated.
17:02 JoeJulian But that would be worth testing.
17:02 cpetersen_ OK.
17:02 JoeJulian Try a heal..full and see if it recreates the gfid hardlinks.
17:05 cpetersen_ So getting back on the netstat subject to verify client connections.  When I do an SS, I can see active connections on the ports shown in gluster v status to each healthy node when node 2 is failed.
17:05 merp_ joined #gluster
17:09 JoeJulian That's more along the lines of what I was expecting. I suspect it's a bug in identifying quorum.
17:11 post-factum JoeJulian: what about reusing tcp ports of removed bricks?
17:11 cpetersen_ =(
17:11 theron joined #gluster
17:12 cpetersen_ The second brick is using a different TCP port than the other two.  Shouldn't matter though right?
17:12 JoeJulian Shouldn't.
17:12 JoeJulian And I don't think that's it either, post-factum.
17:13 post-factum JoeJulian: my question is unrelated to discussed issue, i just would like to know what should I do when port range goes out of permitted by firewall after adding/removing volumes
17:14 JoeJulian I encountered a bug that I've been meaning to reproduce where quorum would fail if I'd manually set the read-subvolume strategy and the specific subvolume wasn't up.
17:14 JoeJulian Oh, well, I use a private network with no firewall so I don't have to care.
17:15 cpetersen_ lol
17:15 post-factum meh
17:15 cpetersen_ Famous last words.
17:15 JoeJulian But... if you were going to script it, I would use the xml output of "gluster volume status" to build the definitions to apply to whatever firewall tool is required.
17:16 jiffin joined #gluster
17:16 Manikandan joined #gluster
17:18 JoeJulian As a matter of fact, work was done recently to add firewalld support. Look and see if that is useful for managing an external firewall.
17:19 cpetersen_ My freaking exports just went the way of the DODO.
17:19 cpetersen_ Freaking hell.
17:21 rcampbel3 joined #gluster
17:28 nathwill joined #gluster
17:28 cpetersen_ Ugh. Moronic.  Apparently when I refreshed my gluster configuration previously, I didn't re-establish passwordless SSH.
17:28 farhorizon joined #gluster
17:28 cpetersen_ Not sure how much of an impact that has but...
17:32 wushudoin| joined #gluster
17:32 RameshN_ joined #gluster
17:38 farhorizon joined #gluster
17:42 theron joined #gluster
17:44 nishanth joined #gluster
17:49 theron joined #gluster
18:01 ovaistariq joined #gluster
18:01 sebamontini joined #gluster
18:01 haomaiwa_ joined #gluster
18:02 cpetersen_ kkeithley: Why when I stop and start the gluster volume do my exports get toasted?
18:02 ovaistar_ joined #gluster
18:02 cpetersen_ Quite literally, the entry in the ganesha.conf to include the export and the actual export file in the exports folder are gone.
18:04 cpetersen_ Is this a bug or intended functionality?
18:04 bennyturns joined #gluster
18:05 RameshN_ joined #gluster
18:07 rafi joined #gluster
18:07 JoeJulian cpetersen_: there's no ganesha hook in the stop hook scripts. The start hook is https://github.com/gluster/glusterfs/blob/master/extras/hook-scripts/start/post/S31ganesha-start.sh
18:07 glusterbot Title: glusterfs/S31ganesha-start.sh at master · gluster/glusterfs · GitHub (at github.com)
18:16 jhyland joined #gluster
18:17 jhyland hey guys! I had a question. If I have multiple glusterFS servers, how do I setup the HA? I mean once the volume is created and everything, do I use a LB to send traffic to the two gfs servers?
18:17 jhyland This is in AWS. I have it setup so I can mount one of the servers directly, that works fine, but most of the tutorials I see end it right there, but if theres multiple servers, im wondering how I should connect to them
18:17 jhyland Is CTDB the popular choice? I havent used it before, but i see it in a few search results
18:17 JoeJulian @mount server
18:17 glusterbot JoeJulian: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.
18:18 JoeJulian ergo, it's HA by design.
18:19 chirino joined #gluster
18:19 cpetersen_ Erm, weird.  My glustershd.log states that my client and server versions are a mismtch.
18:19 cpetersen_ Using Program GlusterFS 3.3
18:19 JoeJulian No it doesn't.
18:20 kanagaraj joined #gluster
18:20 cpetersen_ Server and Client lk-version numbers are not same, reopening the fds
18:20 glusterbot cpetersen_: This is normal behavior and can safely be ignored.
18:20 cpetersen_ OIC.
18:20 JoeJulian booyah!
18:20 cpetersen_ Smart bot.
18:20 cpetersen_ :O
18:20 JoeJulian lk-version.
18:20 JoeJulian Aka, a serialized version tracking of the lock dict.
18:20 cpetersen_ Ahh.. IC.
18:21 cpetersen_ I can't find anything in any logs telling me that the exports were to be deleted.
18:21 cpetersen_ Ugh.
18:22 hchiramm joined #gluster
18:22 JoeJulian My guess was "start" overwrote the config.
18:22 cpetersen_ Possibly, yes.
18:24 cpetersen_ Nope, it was me manually stopping the ganesha service instead of following instruction.
18:24 * cpetersen_ is slapped by JoeJulian with a damp bass
18:24 JoeJulian lol
18:25 cpetersen_ So many componenents... lol
18:27 cpetersen_ It's funny, even with an absolutely fresh setup, I still have these errors in the client logs.
18:27 cpetersen_ http://paste.fedoraproject.org/327957/62520211/
18:27 glusterbot Title: #327957 Fedora Project Pastebin (at paste.fedoraproject.org)
18:27 cpetersen_ I can only assume they are to be ignored.
18:28 JoeJulian I suspect the "Stale file handle" (ESTALE) actually comes from the cache invalidation.
18:28 JoeJulian As it should.
18:29 JoeJulian Probably shouldn't be an " E " if it does, though.
18:29 cpetersen_ I probably shouldn't have cache-invalidation on anyway, correct?
18:29 cpetersen_ I'm not using pNFS.
18:30 JoeJulian No clue.
18:32 cpetersen_ Meh, it still happens even with it off.
18:33 ahino joined #gluster
18:34 JoeJulian Frankly, I don't even know why you have the dht xlator in your graph.
18:34 Wizek_ joined #gluster
18:38 cpetersen_ =)
18:40 jhyland Anyone here have experience with CTDB and Glusterfs?
18:42 jhyland Or do I need it? Im trying to setup some type of HA between the two GFS servers
18:42 sebamontini joined #gluster
18:43 JoeJulian jhyland: please scroll back to where I answered you earlier.
18:43 jhyland oh, sorry
18:43 JoeJulian :)
18:43 jhyland @JoeJulian the '@mount server' ?
18:43 jhyland I know how to mount the server, lol
18:43 JoeJulian Yep
18:43 jhyland But that mounts to one of the two servers..
18:43 jhyland What if the server goes offline?
18:43 JoeJulian Here, read it again.
18:43 JoeJulian @mount server
18:43 glusterbot JoeJulian: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns, or (#2) One caveat is that the clients never learn of any other management peers. If the client cannot communicate with the mount server, that client will not learn of any volume changes.
18:44 jhyland ohhh...
18:44 JoeJulian :)
18:44 jhyland so its me, rethinking shit again, lol.
18:44 jhyland testing it out
18:44 JoeJulian It's a common misconception, which is why it's a factoid.
18:44 julim joined #gluster
18:44 jhyland Whyd you add the @ infront of @mount server ? o.O
18:45 JoeJulian To trigger glusterbot
18:45 JoeJulian So I don't have to type the same stuff over and over again for every new user that misunderstands that bit. :D
18:45 jhyland ok
18:45 JoeJulian I'm inherently lazy.
18:46 JoeJulian s/lazy/efficient/
18:46 glusterbot What JoeJulian meant to say was: I'm inherently efficient.
18:46 jhyland Sign of a good admin
18:46 cpetersen_ rofl
18:47 cpetersen_ JoeJulian:  I recreated my cluster and set the quorum to fixed and count to 2 initially.
18:47 cpetersen_ Things seems to work fine.
18:47 cpetersen_ With 1 node failed.
18:47 cpetersen_ Sorry, brick.
18:47 JoeJulian Interesting
18:47 cpetersen_ >.<
18:48 cpetersen_ I am going to turn brick 2 on again, install the OS, then rinse and repeat.
18:49 jiffin joined #gluster
18:50 Wizek joined #gluster
18:52 jhyland @JoeJulian why do people use CTDB for this then? If GFS handles it already?
18:52 JoeJulian For SMB
18:53 JoeJulian There's still some misguided people that use Windows and think they have to use SMB to integrate it.
18:54 jhyland hahaha
18:54 jhyland indeed
19:01 haomaiwa_ joined #gluster
19:02 farhorizon joined #gluster
19:08 s-hell it's so crazy, two "identical" server. Both connect via georeplication only one working :-( Can anyone help me? https://paste.pcspinnt.de/view/raw/299d6b11
19:08 jiffin joined #gluster
19:13 cpetersen_ JoeJulian:  I installed the VM and shut it down.  I killed brick2.  I attempted to start the VM and it does start.  There are no quorum errors and therefore no write protection.
19:14 cpetersen_ So it seems like there is a bug in the auto quorum type with 3 nodes.
19:14 cpetersen_ s/nodes/brick
19:14 cpetersen_ s/nodes/brick/
19:14 glusterbot What cpetersen_ meant to say was: So it seems like there is a bug in the auto quorum type with 3 brick.
19:14 cpetersen_ s/nodes/bricks/
19:14 glusterbot What cpetersen_ meant to say was: So it seems like there is a bug in the auto quorum type with 3 bricks.
19:14 cpetersen_ AHA!  Glusterbot you are mine now!  You belong to me!
19:15 cpetersen_ Testing all of this reminds of a Walking Dead quote.  "If you have to eat shit, best not to nibble. Bite, chew, swallow, repeat."
19:15 cpetersen_ =]
19:17 Wizek joined #gluster
19:17 voobscout joined #gluster
19:18 cpetersen_ I did not see any split-brain either.
19:19 cpetersen_ Even briefly.
19:33 voobscout joined #gluster
19:35 chirino_m joined #gluster
19:35 farhorizon joined #gluster
19:36 DV joined #gluster
19:44 skylar joined #gluster
19:46 s-hell can anyone please take a look at my error. I can't find any difference
19:46 JoeJulian all that paste says is that something failed at the other end. Need to look at logs on the other end.
19:46 s-hell ok, one moment.
19:47 hackman joined #gluster
19:51 Wizek joined #gluster
19:52 s-hell JoeJulian: Here it is: https://paste.pcspinnt.de/view/raw/4cde84f3
19:55 jwd joined #gluster
19:57 JoeJulian 404
19:58 theron joined #gluster
19:58 s-hell 4040?
19:58 s-hell 404?
19:59 jwd joined #gluster
20:01 haomaiwang joined #gluster
20:04 cpetersen_ JoeJulian: I successfully tested brick2 and host2 failure.  VM moved to host3 and no weird things happened with the quorum on the Gluster.
20:05 cpetersen_ :o
20:05 cpetersen_ Now to test brick1 and host1 failure with VIP floating to brick3
20:05 cpetersen_ I did see the root directory go in to split-brain briefly again, but I am going to ignore it.
20:08 MACscr|lappy joined #gluster
20:11 hagarth joined #gluster
20:12 Wizek joined #gluster
20:44 calavera joined #gluster
20:46 jhyland @JoeJulian Is there a way to speed up how long it takes for glusterfs to switch to other gfs nodes? I setup a cluster with svrA and svrB, then connected a client to svrA, all was well. Then I took svrA offline, and it took a couple minutes for it to work
20:49 JoeJulian @ping timeout
20:49 glusterbot pong
20:49 JoeJulian @ping-timeout
20:49 glusterbot JoeJulian: The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. With an average MTBF of 45000 hours for a server, even just a replica 2 would result in a 42 second MTTR every 2.6 years, or 6 nines of uptime.
20:49 JoeJulian Is six nines not enough?
20:49 jhyland lol
20:50 JoeJulian Or do you just have really shitty servers? ;)
20:50 jhyland Eh, testing it out with AWS
20:50 theron joined #gluster
20:51 JoeJulian Oh, come on. I have friends that work over there. Their stuff isn't *that* bad.
20:51 jhyland I didnt say it was bad :P
20:51 JoeJulian lol
20:51 cpetersen_ :P
20:51 jhyland Im just playing around with gfs, like it so far
20:52 JoeJulian btw... I don't call it that because GFS actually exists.
20:52 jhyland So when I took svrA offline, then messed with a file that was in the nfs, and brought svrA back online, the filereverted back
20:52 jhyland oh
20:52 jhyland ohh, lol
20:52 theron_ joined #gluster
20:53 JoeJulian Also, don't muck with the files on the bricks. Just like you wouldn't dd to a sector on the disk and expect ext4 to know what to do with it, don't mess with the storage that's used as a gluster brick.
20:53 mhulsman joined #gluster
20:53 jhyland No I mean from the client, I didnt do that on the glusterfs server
20:58 post-factum did git blame a little bit, and narrowed "42" value to libglusterfs/src/common-utils.h:41:#define GF_UNIVERSAL_ANSWER 42    /* :O */
20:58 post-factum but that was initial code import, and i couldn't find who the hell invented that
20:58 JoeJulian Life, the universe, and everything.
20:58 post-factum i know, yeah
20:58 post-factum but someone committed that
20:58 JoeJulian That was Avati.
20:59 post-factum nice move for enterprise product ;)
20:59 JoeJulian Again... six nines. It's sufficient.
20:59 cuqa_ joined #gluster
21:00 JoeJulian According to a University of Texas study, the average amount of time it takes a person to plug back in a network cord they tripped over and pulled out of the switch was 41 seconds.
21:01 post-factum oh, i see the reason now
21:01 haomaiwa_ joined #gluster
21:02 post-factum but that should be british scientists and not texas ones
21:05 Wizek joined #gluster
21:05 JoeJulian But then it would have to be in metric.
21:05 cpetersen_ JoeJulian: Simulated (not really) failure of brick1 first, then host1 holding the VM was a success.  Brick1 failed, the VIP floated to brick3 and VMware HA took over after that.  No quorum issues whatsoever.
21:06 JoeJulian Whew
21:06 cpetersen_ So... what have we learned.
21:06 cpetersen_ Specific to VMware, NFS 3 only.
21:07 cpetersen_ NFS 4.1 was tested to work fine in a standard linux client forced to mount as NFS 4.1.
21:07 cpetersen_ Which means KVM would be fine with it.
21:07 cpetersen_ In my 3 node setup, fixing the quorum at 2 prevented the system from locking files such as when using auto quorum type.
21:08 cpetersen_ Beyond learning everything from the ground up, those are the major gotchas.
21:08 cpetersen_ :D
21:10 JoeJulian cpetersen_: Did you already file a bug report for the quorum issue?
21:10 glusterbot https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
21:12 kenhui joined #gluster
21:13 cpetersen_ I did not but I will shortly.
21:15 DV joined #gluster
21:24 Wizek joined #gluster
21:28 skylar joined #gluster
21:29 Wizek_ joined #gluster
21:41 Wizek__ joined #gluster
21:43 sacrelege joined #gluster
21:49 Wizek_ joined #gluster
21:57 Wizek__ joined #gluster
22:01 haomaiwa_ joined #gluster
22:03 MACscr|lappy joined #gluster
22:03 hagarth joined #gluster
22:07 Wizek_ joined #gluster
22:09 DV joined #gluster
22:10 RayTrace_ joined #gluster
22:15 Wizek__ joined #gluster
22:15 farhorizon joined #gluster
22:22 RayTrace_ joined #gluster
22:35 nathwill joined #gluster
22:57 kenhui joined #gluster
22:57 kenhui joined #gluster
22:58 shortdudey123 joined #gluster
22:58 Nebraskka joined #gluster
23:00 Chinorro joined #gluster
23:00 and` joined #gluster
23:00 Arrfab joined #gluster
23:00 tdasilva joined #gluster
23:00 dataio joined #gluster
23:00 ackjewt joined #gluster
23:00 Larsen_ joined #gluster
23:00 moss joined #gluster
23:00 kalzz joined #gluster
23:00 scubacuda joined #gluster
23:00 twisted` joined #gluster
23:00 sjohnsen joined #gluster
23:00 ashka joined #gluster
23:00 Norky joined #gluster
23:00 samppah joined #gluster
23:00 paratai joined #gluster
23:00 mpingu joined #gluster
23:00 Vaelatern joined #gluster
23:00 purpleidea joined #gluster
23:00 bfoster joined #gluster
23:00 yalu joined #gluster
23:00 zoldar joined #gluster
23:00 DJClean joined #gluster
23:00 burn joined #gluster
23:00 rideh joined #gluster
23:00 d4n13L joined #gluster
23:00 tg2 joined #gluster
23:00 Trefex joined #gluster
23:00 steveeJ joined #gluster
23:00 masterzen joined #gluster
23:00 Dasiel joined #gluster
23:00 CyrilPeponnet joined #gluster
23:00 sloop- joined #gluster
23:00 alghost joined #gluster
23:00 fale joined #gluster
23:00 portante joined #gluster
23:00 johnmark joined #gluster
23:00 delhage joined #gluster
23:00 msvbhat joined #gluster
23:00 devilspgd joined #gluster
23:00 Dave joined #gluster
23:00 m0zes joined #gluster
23:00 natarej joined #gluster
23:00 tswartz joined #gluster
23:00 jbrooks joined #gluster
23:00 telmich joined #gluster
23:00 frakt joined #gluster
23:00 scuttle` joined #gluster
23:00 wistof joined #gluster
23:00 muneerse joined #gluster
23:00 social joined #gluster
23:00 morse joined #gluster
23:00 The_Ball joined #gluster
23:00 plarsen joined #gluster
23:00 ninkotech joined #gluster
23:00 sebamontini joined #gluster
23:00 yoavz joined #gluster
23:00 javi404 joined #gluster
23:00 uebera|| joined #gluster
23:00 n-st joined #gluster
23:00 Pintomatic joined #gluster
23:00 badone joined #gluster
23:00 jvandewege joined #gluster
23:00 Ramereth joined #gluster
23:00 bio_ joined #gluster
23:00 ccha2 joined #gluster
23:00 glusterbot joined #gluster
23:00 sadbox joined #gluster
23:00 suliba joined #gluster
23:00 unforgiven512 joined #gluster
23:00 valkyr1e joined #gluster
23:00 PotatoGim joined #gluster
23:00 sghatty_ joined #gluster
23:00 sankarshan_away joined #gluster
23:00 Nakiri__ joined #gluster
23:00 anoopcs joined #gluster
23:00 coredump|br joined #gluster
23:00 chirino_m joined #gluster
23:00 haomaiwa_ joined #gluster
23:00 yawkat joined #gluster
23:00 al joined #gluster
23:00 ChrisHolcombe joined #gluster
23:00 kbyrne joined #gluster
23:00 tom[] joined #gluster
23:00 p8952 joined #gluster
23:00 mdavidson joined #gluster
23:00 honzik666 joined #gluster
23:00 edualbus joined #gluster
23:00 Peppard joined #gluster
23:00 martinetd joined #gluster
23:00 JPaul joined #gluster
23:00 post-factum joined #gluster
23:00 jwang__ joined #gluster
23:00 jermudgeon joined #gluster
23:00 Chr1st1an joined #gluster
23:00 sc0 joined #gluster
23:00 billputer joined #gluster
23:00 lh_ joined #gluster
23:00 liewegas joined #gluster
23:00 sagarhani joined #gluster
23:00 jockek joined #gluster
23:01 DV joined #gluster
23:01 skylar joined #gluster
23:01 calavera joined #gluster
23:01 samikshan joined #gluster
23:01 s-hell joined #gluster
23:01 tru_tru joined #gluster
23:01 ws2k3 joined #gluster
23:01 dron23 joined #gluster
23:01 virusuy joined #gluster
23:01 fyxim joined #gluster
23:01 marbu joined #gluster
23:01 mowntan joined #gluster
23:01 siel joined #gluster
23:01 dgandhi joined #gluster
23:01 Guest62935 joined #gluster
23:01 lkoranda joined #gluster
23:01 haomaiwa_ joined #gluster
23:02 kenhui joined #gluster
23:02 hackman joined #gluster
23:02 python_lover joined #gluster
23:02 shruti joined #gluster
23:02 nhayashi joined #gluster
23:02 cyberbootje joined #gluster
23:02 john51 joined #gluster
23:02 Iouns joined #gluster
23:02 dastar joined #gluster
23:02 csaba joined #gluster
23:02 the-me joined #gluster
23:02 Kins joined #gluster
23:02 edong23 joined #gluster
23:02 ninkotech_ joined #gluster
23:02 atrius joined #gluster
23:02 Ulrar joined #gluster
23:02 marlinc joined #gluster
23:03 hagarth joined #gluster
23:03 ovaistar_ joined #gluster
23:03 wushudoin| joined #gluster
23:03 F2Knight joined #gluster
23:03 pdrakeweb joined #gluster
23:03 _fortis joined #gluster
23:03 bhuddah joined #gluster
23:03 malevolent joined #gluster
23:03 inodb joined #gluster
23:03 owlbot joined #gluster
23:03 baoboa joined #gluster
23:03 XpineX joined #gluster
23:03 Telsin joined #gluster
23:03 rastar joined #gluster
23:03 Vaizki joined #gluster
23:03 lalatenduM joined #gluster
23:03 Plysdyret joined #gluster
23:03 SpeeR joined #gluster
23:03 rossdm joined #gluster
23:03 wiza joined #gluster
23:03 troj joined #gluster
23:03 dries joined #gluster
23:03 cristian joined #gluster
23:03 [o__o] joined #gluster
23:03 necrogami joined #gluster
23:03 misc joined #gluster
23:03 sac joined #gluster
23:03 PaulePanter joined #gluster
23:03 ndevos joined #gluster
23:03 JoeJulian joined #gluster
23:03 abyss_ joined #gluster
23:03 decay joined #gluster
23:03 lanning joined #gluster
23:03 Logos01 joined #gluster
23:03 swebb joined #gluster
23:03 eljrax joined #gluster
23:03 oliva joined #gluster
23:03 semiosis joined #gluster
23:03 kmmndr joined #gluster
23:03 crashmag joined #gluster
23:03 k-ma joined #gluster
23:03 john51 joined #gluster
23:03 jhyland joined #gluster
23:03 bennyturns joined #gluster
23:03 harish joined #gluster
23:03 Gugge joined #gluster
23:03 v12aml joined #gluster
23:04 PaulePanter joined #gluster
23:06 MACscr|lappy joined #gluster
23:06 ahino joined #gluster
23:06 merp_ joined #gluster
23:06 17WAADGCT joined #gluster
23:06 neofob joined #gluster
23:06 dblack joined #gluster
23:06 petan joined #gluster
23:06 ghenry joined #gluster
23:06 papamoose joined #gluster
23:06 mzink_gone joined #gluster
23:06 xMopxShell joined #gluster
23:07 rGil joined #gluster
23:07 mzink_gone joined #gluster
23:07 john51 joined #gluster
23:08 mlhess joined #gluster
23:09 mlhess joined #gluster
23:10 sc0 joined #gluster
23:11 javi404 joined #gluster
23:11 yoavz joined #gluster
23:15 javi404 joined #gluster
23:15 yoavz joined #gluster
23:17 sloop joined #gluster
23:18 primusinterpares joined #gluster
23:18 mzink_gone joined #gluster
23:18 sc0 joined #gluster
23:25 PotatoGim joined #gluster
23:30 hagarth joined #gluster
23:30 twisted` joined #gluster
23:37 john51_ joined #gluster
23:41 calavera joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary