Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-03-20

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:34 robo joined #gluster
00:36 tokik joined #gluster
00:38 Elico left #gluster
00:42 Durzo joined #gluster
00:43 Durzo hi all, wondering if gluster+kvm (libgfapi) is stable yet? all the info on the net i can find are old and mention "upcoming 3.4 beta".. since we are nearly 3.5 i hope its stable for production use ??
00:44 coredump joined #gluster
00:52 yinyin joined #gluster
00:55 neurodrone joined #gluster
00:57 nightwalk joined #gluster
01:01 chirino_m joined #gluster
01:02 * Durzo pokes JoeJulian
01:33 jobewan joined #gluster
01:37 discretestates joined #gluster
01:43 nightwalk joined #gluster
01:45 mattappe_ joined #gluster
01:54 discretestates joined #gluster
01:57 mattappe_ joined #gluster
01:57 nightwalk joined #gluster
01:58 yinyin joined #gluster
01:59 DV joined #gluster
02:04 hagarth joined #gluster
02:06 m0zes joined #gluster
02:06 harish joined #gluster
02:08 mattappe_ joined #gluster
02:18 nightwalk joined #gluster
02:21 pdrakeweb joined #gluster
02:31 kam270 joined #gluster
02:50 bharata-rao joined #gluster
02:52 Durzo according to http://www.gluster.org/community/d​ocumentation/index.php/Planning35, 3.5 should have been GA by end of January... is there some updated roadmap ?
02:56 kam270__ joined #gluster
03:00 glusterbot New news from newglusterbugs: [Bug 991084] No way to start a failed brick when replaced the location with empty folder <https://bugzilla.redhat.com/show_bug.cgi?id=991084>
03:02 chirino joined #gluster
03:06 rshade98 joined #gluster
03:07 discretestates joined #gluster
03:07 nightwalk joined #gluster
03:16 yinyin joined #gluster
03:30 jclift Durzo: We'll probably have it out within 2 weeks
03:30 jclift Durzo: We've been releasing betas for a while now, and have chased down almost all of the outstanding issues
03:31 jclift Durzo: When they're fixed, it'll be released
03:32 jclift Durzo: It's mentioned in the conversation of yesterday's weekly Community Meeting if that's your kind of thing to look through: http://meetbot.fedoraproject.org/g​luster-meeting/2014-03-19/gluster-​meeting.2014-03-19-15.01.log.html
03:32 glusterbot Title: #gluster-meeting log (at meetbot.fedoraproject.org)
03:33 jclift Hope that helps :)
03:33 * jclift heads off to bed
03:35 RameshN joined #gluster
03:35 Durzo i saw there was a mem leak in beta4, has that been fixed jclift ?
03:36 itisravi joined #gluster
03:38 * Durzo reading meeting log
03:44 discrete_ joined #gluster
03:50 kanagaraj joined #gluster
03:53 m0zes joined #gluster
03:57 nightwalk joined #gluster
04:02 hagarth joined #gluster
04:05 shubhendu joined #gluster
04:05 chirino joined #gluster
04:18 sahina joined #gluster
04:20 haomaiwa_ joined #gluster
04:27 mohankumar joined #gluster
04:29 sks joined #gluster
04:30 dusmant joined #gluster
04:33 rahulcs joined #gluster
04:37 prasanthp joined #gluster
04:48 vpshastry joined #gluster
04:48 nightwalk joined #gluster
04:48 yinyin joined #gluster
04:51 haomai___ joined #gluster
04:56 davinder joined #gluster
04:57 ndarshan joined #gluster
04:59 discretestates joined #gluster
05:00 bala joined #gluster
05:00 glusterbot New news from newglusterbugs: [Bug 1051993] Force argument is ambiguous <https://bugzilla.redhat.co​m/show_bug.cgi?id=1051993>
05:01 ravindran joined #gluster
05:04 chirino_m joined #gluster
05:07 kdhananjay joined #gluster
05:13 ravindran joined #gluster
05:15 [o__o] left #gluster
05:18 [o__o] joined #gluster
05:19 GabrieleV joined #gluster
05:21 [o__o] left #gluster
05:23 deepakcs joined #gluster
05:24 [o__o] joined #gluster
05:27 aravindavk joined #gluster
05:27 [o__o] left #gluster
05:31 [o__o] joined #gluster
05:33 vimal joined #gluster
05:34 [o__o] left #gluster
05:37 [o__o] joined #gluster
05:40 discretestates joined #gluster
05:45 nightwalk joined #gluster
05:46 vpshastry joined #gluster
05:47 vpshastry joined #gluster
05:57 haomaiwang joined #gluster
05:57 ppai joined #gluster
06:10 nshaikh joined #gluster
06:13 benjamin_____ joined #gluster
06:14 gmcwhistler joined #gluster
06:19 davinder2 joined #gluster
06:27 rahulcs joined #gluster
06:30 lalatenduM joined #gluster
06:31 micu joined #gluster
06:38 raghu joined #gluster
06:39 ricky-ti1 joined #gluster
06:40 kanagaraj joined #gluster
06:41 nightwalk joined #gluster
06:46 prasanthp joined #gluster
06:48 vpshastry joined #gluster
06:48 kanagaraj joined #gluster
06:56 ndarshan joined #gluster
07:05 slayer192 joined #gluster
07:14 gmcwhist_ joined #gluster
07:14 ngoswami joined #gluster
07:17 gmcwhistler joined #gluster
07:17 vpshastry joined #gluster
07:18 psharma joined #gluster
07:19 gmcwhistler joined #gluster
07:30 XATRIX joined #gluster
07:37 andreask joined #gluster
07:39 cyberbootje joined #gluster
07:44 ekuric joined #gluster
07:44 nightwalk joined #gluster
07:52 prasanthp joined #gluster
07:52 ctria joined #gluster
07:58 ekuric joined #gluster
07:58 vpshastry joined #gluster
08:01 eseyman joined #gluster
08:08 pdrakeweb joined #gluster
08:17 mattappe_ joined #gluster
08:17 cjanbanan joined #gluster
08:21 badone_ joined #gluster
08:26 keytab joined #gluster
08:36 fsimonce joined #gluster
08:38 chirino joined #gluster
08:38 eseyman joined #gluster
08:40 Pavid7 joined #gluster
08:46 ProT-0-TypE joined #gluster
08:50 wgao joined #gluster
08:56 harish joined #gluster
09:00 nightwalk joined #gluster
09:01 glusterbot New news from newglusterbugs: [Bug 1022510] GlusterFS client crashes during add-brick and rebalance <https://bugzilla.redhat.co​m/show_bug.cgi?id=1022510>
09:01 prasanthp joined #gluster
09:07 X3NQ joined #gluster
09:12 vpshastry joined #gluster
09:12 FarbrorLeon joined #gluster
09:23 sks joined #gluster
09:25 raghu joined #gluster
09:25 Norky joined #gluster
09:26 bala joined #gluster
09:29 shylesh joined #gluster
09:31 liquidat joined #gluster
09:35 jmarley joined #gluster
09:40 brosner joined #gluster
09:45 tokik joined #gluster
09:53 tokik joined #gluster
09:53 nightwalk joined #gluster
09:58 Pavid7 joined #gluster
10:08 pdrakeweb joined #gluster
10:16 monotek did somebofy allready tried glusterfs with libgfapi / qemu an got an libvirt machine started?
10:16 monotek for me only qemu-img works...
10:16 ndevos sure, it works for me on CloudStack
10:19 monotek could you show me your libvirt vm xml file?
10:19 monotek currently i get this error when trying to start:
10:19 monotek 2014-03-19 18:48:40.000+0000: 22004: warning : qemuDomainObjBeginJobInternal:1119 : Cannot start job (modify, none) for domain glustertest; current job is (modify, none) owned by (22007, 0)
10:19 monotek 2014-03-19 18:48:40.000+0000: 22004: error : qemuDomainObjBeginJobInternal:1123 : Timed out during operation: cannot acquire state change lock
10:20 monotek this is the disk part of my xml: http://pastebin.com/jGFWgRaR
10:20 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
10:21 mohankumar joined #gluster
10:21 monotek http://paste.ubuntu.com/7124521/
10:21 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
10:31 ndevos monotek: http://paste.fedoraproject.org/86984/31148013 is one of the vms
10:31 glusterbot Title: #86984 Fedora Project Pastebin (at paste.fedoraproject.org)
10:33 ndevos monotek: you may need to set some attributes to allow non-root access, see http://www.ovirt.org/Features/GlusterFS_​Storage_Domain#Important_Pre-requisites
10:33 glusterbot Title: Features/GlusterFS Storage Domain (at www.ovirt.org)
10:47 raghug joined #gluster
10:54 vpshastry joined #gluster
10:57 rwheeler joined #gluster
10:57 jbustos joined #gluster
10:58 jiffe98 so this has been a less than stellar upgrade
10:58 jiffe98 upgraded the last node, it sees no peers
11:01 nightwalk joined #gluster
11:04 spandit joined #gluster
11:07 Norky_ joined #gluster
11:09 jiffe98 nm, that was my mistake, copied the /local* gluster files to gluster instead of glusterd
11:10 prasanthp joined #gluster
11:11 jiffe98 but this node is not connecting either, same as the last node I upgraded yesterday
11:11 jiffe98 so I have 2 nodes connected to each other and the other 2 nodes rejected by everyone
11:12 abyss^ I did something like this: http://pastebin.ca/2668142 but when I try to ls gluster mount point on client then ls freezes. Logs show that everything is ok, any ideas where to lookin?
11:12 glusterbot Title: pastebin - gluster - post number 2668142 (at pastebin.ca)
11:15 abyss^ and get very high load (CPU)
11:15 ctria joined #gluster
11:15 rahulcs_ joined #gluster
11:17 skryzhny joined #gluster
11:20 mattappe_ joined #gluster
11:20 abyss^ (on client)
11:29 mattappe_ joined #gluster
11:37 vpshastry joined #gluster
11:41 Slash joined #gluster
11:43 gmcwhistler joined #gluster
11:44 abyss^ it's happens only when I try ls on mounted directory
11:44 RicardoSSP joined #gluster
11:44 RicardoSSP joined #gluster
11:56 kkeithley joined #gluster
11:57 nightwalk joined #gluster
12:00 kkeithley1 joined #gluster
12:02 aravindavk joined #gluster
12:04 itisravi joined #gluster
12:08 vpshastry joined #gluster
12:08 jiffe98 was definitely something not syncing between the peers, copied the volume configs to the non working peers and they connect now
12:09 pdrakeweb joined #gluster
12:14 monotek nevdos... thnaks for the hint... tried to set this values in my volume...
12:14 monotek gluster volume info gv8 -> http://paste.ubuntu.com/7124923/
12:14 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
12:14 pdrakeweb joined #gluster
12:14 monotek does not work...
12:14 prasanthp joined #gluster
12:15 ndevos monotek: if you changed server.allow-insecure on the volume, you have to stop and start the volume to apply that option
12:15 monotek ah.. ok... thanks... will try....
12:15 ndevos monotek: also, you have to make sure that you have restarted glusterd after setting that similar option in the glusterd.vol
12:17 monotek if set the values via commandline:
12:17 monotek gluster volume set gv8 server.allow-insecure on
12:17 monotek gluster volume set gv8 rpc-auth-allow insecure
12:17 monotek ok... have to try this later... cant stop the volume now :-/
12:18 ndevos monotek: no, the 'rpc-auth-allow insecure' is an option for glusterd, it should be set in /etc/glusterfs/glusterd.vol
12:18 ctria joined #gluster
12:18 dusmant joined #gluster
12:23 qdk joined #gluster
12:24 ppai joined #gluster
12:28 monotek thanks for your help. no ui have something where i can start... think i have to try this on a weekend....
12:29 saurabh joined #gluster
12:32 glusterbot New news from newglusterbugs: [Bug 1078847] DHT- mkdir fails and gives error, 'invalid argument' if hashed sub-volume is down <https://bugzilla.redhat.co​m/show_bug.cgi?id=1078847>
12:32 kanagaraj joined #gluster
12:33 recidive joined #gluster
12:35 kiwnix joined #gluster
12:38 rfortier1 joined #gluster
12:40 ndevos monotek: good luck, and please let us know if it does (not) work
12:40 monotek i will :-)
12:47 nightwalk joined #gluster
12:49 shyam joined #gluster
12:51 dusmant joined #gluster
12:53 edward1 joined #gluster
12:53 raghug joined #gluster
12:54 kiwnix joined #gluster
12:59 hagarth joined #gluster
13:02 ricky-ti1 joined #gluster
13:03 kam270 joined #gluster
13:05 jag3773 joined #gluster
13:09 pk1 joined #gluster
13:10 tdasilva joined #gluster
13:13 robo joined #gluster
13:18 aravindavk joined #gluster
13:19 benjamin_____ joined #gluster
13:22 sroy joined #gluster
13:23 kiwnix joined #gluster
13:30 dewey joined #gluster
13:30 theron joined #gluster
13:30 primechuck joined #gluster
13:30 xymox joined #gluster
13:32 kam270 joined #gluster
13:33 ppai joined #gluster
13:36 shubhendu joined #gluster
13:38 nightwalk joined #gluster
13:42 xymox joined #gluster
13:45 P0w3r3d joined #gluster
13:50 kam270 joined #gluster
13:51 seapasulli joined #gluster
13:51 failshell joined #gluster
13:53 xymox joined #gluster
13:53 saravanakumar1 joined #gluster
13:54 saravanakumar1 Hello everyone
13:54 saravanakumar1 Hello everyone
13:54 saravanakumar1 i am new to glusterfs
13:54 saravanakumar1 i am really confused with hadoop and gluster can someone pls help me to clarify
13:54 saravanakumar1 thanks
13:55 coredump joined #gluster
13:57 discretestates joined #gluster
13:57 isaacabo joined #gluster
13:58 isaacabo Good morning
13:58 pk1 left #gluster
13:58 isaacabo i have a question, wondering is this is normal after rebalancing a gluster
13:58 isaacabo yesterday we add a 8TB brick to our 12TB brick, the rebalance finished
13:59 X3NQ_ joined #gluster
13:59 isaacabo [root@store1 glusterfs]# gluster volume rebalance shares status
13:59 isaacabo Node Rebalanced-files          size       scanned      failures       skipped         status run time in secs
13:59 isaacabo ---------      -----------   -----------   -----------   -----------   -----------   ------------   --------------
13:59 isaacabo localhost           442884       226.8GB       1344384             0             0      completed        140901.00
13:59 isaacabo store2.office.mentel.com                1       40Bytes        901497             0             2      completed         39461.00
13:59 lalatenduM @pastebin | isaacabo
13:59 japuzzo joined #gluster
13:59 lalatenduM @pastebin
13:59 glusterbot lalatenduM: I do not know about 'pastebin', but I do know about these similar topics: 'paste', 'pasteinfo'
13:59 lalatenduM @paste
13:59 glusterbot lalatenduM: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
13:59 DV joined #gluster
13:59 lalatenduM @paste | isaacabo
14:00 isaacabo ok, let me try that]
14:00 lalatenduM saravanakumar1, sorry didn't get the question. do u want to know the diff between the two?
14:01 isaacabo no, the question is it normal that the original brick still full?
14:01 saravanakumar1 lalatenduM yes. Do we have any docs to understand
14:02 xymox joined #gluster
14:02 robo joined #gluster
14:04 lalatenduM saravanakumar1, I am sure there will be lot on the internet , I think you are talking about "Hadoop Distributed File System" (not Hadoop)  VS "GLusterFS"
14:05 saravanakumar1 lalatenduM: yes you are right am asking about hdfs and glusterfs
14:05 ndk joined #gluster
14:06 isaacabo http://ur1.ca/gw27d -> http://paste.fedoraproject.org/87046/24340139
14:06 glusterbot Title: #87046 Fedora Project Pastebin (at ur1.ca)
14:06 isaacabo thx for this tip
14:06 lalatenduM saravanakumar1, u r question is this http://www.quora.com/What-are-the-pros​-and-cons-for-using-HDFS-vs-Gluster-es​pecially-for-Hadoop/answer/Jeff-Darcy
14:06 glusterbot Title: Jeff Darcys answer to What are the pros and cons for using HDFS vs. Gluster (especially for Hadoop)? - Quora (at www.quora.com)
14:07 isaacabo but the original brick is still nearly full
14:07 lalatenduM isaacabo, it is ok to see skip files in rebalance status
14:08 lalatenduM isaacabo, I think a force rebalance will help u
14:08 xymox joined #gluster
14:08 saravanakumar1 thanks lalatenduM and  glusterbot
14:09 kam270 joined #gluster
14:09 isaacabo What do a force rebalance?
14:11 rwheeler joined #gluster
14:16 xymox joined #gluster
14:20 lmickh joined #gluster
14:20 lalatenduM isaacabo, the two files got skipped because when you were doing rebalance , the new brick has less available size(total available size)  than the old brick, however you might have enough space on the new brick to move the files, even if the available size is less than the old brick's available size
14:21 lalatenduM isaacabo, with force these files will be moved to correct brick
14:21 jiffe98 so after my upgrade to 3.4.2 I have one volume that seems to be running extremely slow, like several minutes to ls a directory with 3 entries
14:22 DV joined #gluster
14:22 lalatenduM jiffe98, check volume status, is self heal is running o this?
14:23 lalatenduM jiffe98, also you should check the memory usage of the server
14:23 isaacabo Ok, thx. Then all the data will be distributed between the two bricks?
14:24 xymox joined #gluster
14:26 kanagaraj joined #gluster
14:29 jiffe98 lalatenduM: all of the servers have at least 2G of memory free, nothing in swap, volume status shows no active tasks but volume heal VOLNAME info just hangs
14:32 isaacabo so, should i issue this command gluster volume rebalance VOLNAME migrate-data start ?
14:32 lalatenduM jiffe98, may be it is taking time to comeup , and a side effect of the issue
14:32 rpowell joined #gluster
14:33 xymox joined #gluster
14:33 isaacabo or gluster volume rebalance VOLNAME start force ?
14:34 lalatenduM isaacabo, here is the usage "volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force]"  , before you run the command, which version of gluster u r using?
14:34 nightwalk joined #gluster
14:34 isaacabo glusterfs 3.4.2 built on Jan  3 on CentOs
14:35 lalatenduM isaacabo, ok
14:35 lalatenduM @rebalance
14:35 glusterbot lalatenduM: I do not know about 'rebalance', but I do know about these similar topics: 'replace'
14:36 _dist joined #gluster
14:37 jiffe98 lalatenduM: its been about 3 hours :\
14:37 jiffe98 that volume isn't very large
14:38 lalatenduM jiffe98, may be something went bad, is it ur production data?
14:39 jiffe98 lalatenduM: it is
14:39 jiffe98 3 volume seem to be fine, just seems to be this one
14:39 jiffe98 volumes
14:40 robo joined #gluster
14:40 jiffe98 I noticed it has cluster.lookup-unhashed: off set whereas the others don't
14:41 isaacabo Ok, thank u.
14:42 lalatenduM jiffe98, not sure abt that vol set option
14:43 xymox joined #gluster
14:45 jiffe98 3.3 seemed a whole lot more stable
14:46 Lethalman joined #gluster
14:47 Lethalman hi, has multi-master geo replica been implemented already in 3.5 beta?
14:51 xymox joined #gluster
14:54 isaacabo thank you all for the info
14:55 lalatenduM isaacabo, u r welcome!
14:55 lalatenduM Lethalman, msvbhat is the person you are looking for :)
14:55 Lethalman lalatenduM, thanks
14:59 B21956 joined #gluster
15:01 xymox joined #gluster
15:03 coredump_ joined #gluster
15:03 benjamin_____ joined #gluster
15:07 sijis joined #gluster
15:08 xymox joined #gluster
15:09 ndk` joined #gluster
15:12 dbruhn joined #gluster
15:12 robo joined #gluster
15:13 msvbhat Lethalman: No, multimaster is not yet implemented.
15:14 Lethalman msvbhat, thanks
15:14 kam270 joined #gluster
15:14 sijis i'm trying to mount a volume localy to test, but i see a 'failed to get volume file' from server' (http://paste.fedoraproject.org/87074/13953284/) any suggestions?
15:14 glusterbot Title: #87074 Fedora Project Pastebin (at paste.fedoraproject.org)
15:15 msvbhat sijis: You should give volume name for mount
15:15 msvbhat sijis: This should be your command. sudo mount -t glusterfs localhost:gv0 /mnt
15:15 sijis msvbhat: mount -t glusterfs gv0 /mnt ?
15:16 msvbhat sijis: NO, It's "mount -t glusterfs <Any server in cluster>:<volume name> <mount point>
15:17 msvbhat In your case mount -t glusterfs localhost:gv0 /mnt
15:17 sijis msvbhat: gotcha. that worked. so it should be the same for a client too, correct?
15:18 _dist personally I like using localhost incase the mount happens while the one you point to is down. If the localhost brick is unhealthy it'll just find a good brick
15:18 xymox joined #gluster
15:19 msvbhat sijis: --> so it should be the same for a client too, correct? <-- I did not understand this.
15:19 rahulcs joined #gluster
15:20 msvbhat sijis: You mount the gluster volume from client. But the same node can be server and client as well.
15:20 sijis msvbhat: on a non glusterfs server .. it would be mount -t glusterfs dil-vm-gfs-02:gv0 /mnt/
15:20 msvbhat sijis: Yes...
15:20 sijis msvbhat: right. i understand the server can also be a client
15:21 semiosis ,,(mount server)
15:21 glusterbot The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
15:21 semiosis fwiw
15:22 diegows joined #gluster
15:23 sijis semiosis: that's what i understood.. but i was using epel's 3.2x version and yet when i rebooted one of the 2 servers, the client (that used node1) was never able to get data.. until node1 was up.
15:23 jag3773 joined #gluster
15:24 xymox joined #gluster
15:25 Staples84 joined #gluster
15:25 nueces_ joined #gluster
15:27 sijis im using 3.4.2 right now
15:31 jmarley joined #gluster
15:31 jmarley joined #gluster
15:32 msvbhat sijis: What was your volume configuration?
15:33 msvbhat sijis: You are mounting via gluster native mount right?
15:34 xymox joined #gluster
15:35 sks joined #gluster
15:36 sijis msvbhat: same as the one i posted
15:36 sijis msvbhat: and native mount too
15:37 discretestates joined #gluster
15:43 xymox joined #gluster
15:44 tdasilva left #gluster
15:47 kam270 joined #gluster
15:51 xymox joined #gluster
16:01 robo joined #gluster
16:02 xymox joined #gluster
16:03 kam270 joined #gluster
16:07 vpshastry joined #gluster
16:07 Peanut__ Hi, I have a question about adding a client. I've got two machine running a Gluster (3.4.1), which works fine. I've now created a second volume on them (replicate), and I want to mount that from an external box. "mount -t glusterfs cl0:/gv1 /mnt" works without error, and I can write to it. But the data does not end up on the bricks, and is gone after a reboot. What am I missing?
16:09 xymox joined #gluster
16:11 bennyturns joined #gluster
16:12 ndevos Peanut__: I've only seen that happen when the filesystem for the bricks were not mounted automatically, or were not mounted when the volume was created
16:12 kam270 joined #gluster
16:13 discretestates joined #gluster
16:13 Peanut__ Niels: they're surely mounted on the servers, and were before I created the volume. Not sure what you mean with 'mounted automatically' - there's not been a reboot since I created his new volume.
16:18 xymox joined #gluster
16:19 rfortier1 joined #gluster
16:21 Peanut Interesting - if I mount the new volume on one of the gluster servers, it does work as expected.
16:21 Peanut Is 3.4.2 incompatible with 3.4.1 ?
16:22 Mo_ joined #gluster
16:22 Peanut Ah well, now it magically works from the client, too.
16:24 shyam joined #gluster
16:26 TNTony joined #gluster
16:26 xymox joined #gluster
16:26 hagarth joined #gluster
16:30 msvbhat sijis: is it working in 3.4.2? If mountpoint is not accessible, can you open a bug?
16:31 criticalhammer Is there documentation on how gluster utilizes the build in NFS server?
16:32 criticalhammer It seems it nfs is used but I can't seem to find anything on how or why.
16:32 kam270 joined #gluster
16:33 shyam left #gluster
16:33 awheeler_ joined #gluster
16:35 xymox joined #gluster
16:42 sijis msvbhat: i'm checking on 3.4.2. with the same test.
16:42 sijis i iwll open a bug if that fails
16:44 xymox joined #gluster
16:44 recidive joined #gluster
16:46 chirino_m joined #gluster
16:53 xymox joined #gluster
16:56 kam270 joined #gluster
17:00 theron joined #gluster
17:03 xymox joined #gluster
17:05 sijis msvbhat: still no luck. maybe my test is flawed? i have 2 nodes (gfs-01/02). the client (app-11) mounted it via mount -t glusterfs gfs-01:gv0 /mnt/test. on app-11 i ran a simple lopos htat copies /var/log/messages 10000 times to /mtn/test. after about 1 min, i reboot gfs-01. in the meantime, i'm running ls -l | wc -l on /export/gv0 on gfs-02.. but as soon as gfs-01 becomes unpingable.. the number of files does not increase
17:05 sijis it begins to increase on gfs-02 the moment gfs01 is pingable again
17:07 sijis loop - for i in `seq -w 1 10000`; do sudo cp -rp /var/log/messages /mnt/test/copy-test-$i; done
17:10 hagarth joined #gluster
17:11 xymox joined #gluster
17:14 isaacabo joined #gluster
17:15 dusmant joined #gluster
17:17 zerick joined #gluster
17:18 jobewan joined #gluster
17:20 mohankumar joined #gluster
17:20 xymox joined #gluster
17:22 kam270 joined #gluster
17:26 diegows joined #gluster
17:29 xymox joined #gluster
17:34 vpshastry joined #gluster
17:37 xymox joined #gluster
17:39 rfortier joined #gluster
17:46 kam270 joined #gluster
17:46 coredum__ joined #gluster
17:47 lalatenduM joined #gluster
17:47 xymox joined #gluster
17:49 primechu_ joined #gluster
17:51 zaitcev joined #gluster
17:57 xymox joined #gluster
18:03 kam270 joined #gluster
18:03 elyograg joined #gluster
18:05 xymox joined #gluster
18:05 elyograg My NFS server crashed today.  Nothing useful that I could see in the logs other than a strange error message that has been plaguing us for a while.  And now on one of the brick servers, I am seeing this happening over and over in the etc-glusterfs-glusterd.vol.log: I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
18:06 elyograg gluster NFS server crashed, that is.
18:06 robo joined #gluster
18:06 elyograg actually it might be more than the one brick server.  i've only looked at one.
18:06 elyograg joined #gluster
18:15 xymox joined #gluster
18:21 kam270 joined #gluster
18:25 xymox joined #gluster
18:35 kam270 joined #gluster
18:41 chirino joined #gluster
18:44 xymox joined #gluster
18:50 slayer192 joined #gluster
18:55 JoeJulian elyograg: All that says is that something is disconnecting (tcp FIN). You may want to look at a tcpdump to find out what's doing that.
18:56 xymox joined #gluster
18:58 kam270 joined #gluster
19:02 jmarley joined #gluster
19:02 jmarley joined #gluster
19:07 xymox joined #gluster
19:08 kam270 joined #gluster
19:16 nightwalk joined #gluster
19:18 robos joined #gluster
19:30 xymox joined #gluster
19:34 kam270 joined #gluster
19:43 elyograg tcpdump for FIN packets turns up nothing.  I verified it was working right by opening and closing an ssh session while sniffing.
19:44 johnbot11 joined #gluster
19:45 tdasilva joined #gluster
19:45 nightwalk joined #gluster
19:46 vu joined #gluster
19:46 recidive joined #gluster
19:50 chirino_m joined #gluster
19:51 xymox joined #gluster
19:55 kam270 joined #gluster
20:05 xymox joined #gluster
20:09 Pavid7 joined #gluster
20:10 mtanner_ joined #gluster
20:10 kaptk2 joined #gluster
20:12 ThatGraemeGuy joined #gluster
20:12 xymox joined #gluster
20:15 JoeJulian elyograg: I wonder if closing the named pipe hits the same code path. I've not worked with sockets using named pipes, but I suspect it does.
20:21 xymox joined #gluster
20:25 kam270 joined #gluster
20:29 wrale joined #gluster
20:31 xymox joined #gluster
20:35 ricky-ti1 joined #gluster
20:38 kam270 joined #gluster
20:40 xymox joined #gluster
20:49 xymox joined #gluster
20:49 kam270 joined #gluster
20:57 robo joined #gluster
20:57 kam270 joined #gluster
20:57 xymox joined #gluster
20:59 andreask joined #gluster
20:59 robo joined #gluster
21:00 nightwalk joined #gluster
21:01 tdasilva left #gluster
21:04 sijis is this a valid failover test? i have 2 nodes (gfs-01/02). the client (app-11) mounted it via mount -t glusterfs gfs-01:gv0 /mnt/test. on app-11 i ran a simple loop that copies /var/log/messages 10000 times to /mnt/test. after about 1 min, i reboot gfs-01. in the meantime, i'm running ls -l | wc -l on /export/gv0 on gfs-02.. but as soon as gfs-01 becomes unpingable.. the number of files created on gfs-02 does not increase.
21:05 semiosis clients will hang for ping-timeout when a server disconnects abruptly
21:05 xymox joined #gluster
21:05 semiosis graceful shutdown does not cause the ping timeout
21:05 semiosis pulling a cable does
21:05 primechuck joined #gluster
21:05 sijis semiosis: so i should unplug the cable on it?
21:06 sijis instead
21:06 semiosis idk
21:06 semiosis probably not?
21:06 semiosis check the log file for the client mount doing the writing
21:06 semiosis log file path should be /var/log/glusterfs/the-mount-point.log
21:06 semiosis it should say what it's doing about the failover
21:08 sijis ok. let me look at that
21:08 andreask joined #gluster
21:14 badone_ joined #gluster
21:14 xymox joined #gluster
21:15 kam270 joined #gluster
21:16 sijis semiosis: interesting. it is working when i disconnect the cable
21:17 sijis i wonder if the reboot time is much faster than the timeout allowed
21:17 sijis so the vm comes up before the timeout is reached
21:17 semiosis the server closes the connection when it reboots gracefully, so clients don't wait for it
21:18 JoeJulian Or at least it's supposed to. Seems to be broken.
21:18 semiosis anyway, too busy for guessing.  if you want to pastie a log file i'll take a look.  make sure it goes all the way from the start of the mount until the failure
21:18 semiosis JoeJulian: orly?!
21:18 sijis semiosis: log from client or server.. or both?
21:19 JoeJulian I submitted a patch, but it seems to have gone stale.
21:19 semiosis sijis: client log
21:19 sijis semiosis: ok.
21:20 JoeJulian http://review.gluster.org/#/c/7195/
21:20 glusterbot Title: Gerrit Code Review (at review.gluster.org)
21:21 semiosis JoeJulian: wait a sec, i thought it was the SIGKILL sent to the glusterfsd procs that caused the graceful disconnect
21:21 semiosis JoeJulian: debuntu doesnt have any glusterfsd kill initscript
21:21 JoeJulian It would have to happen before the network is stopped.
21:21 semiosis JoeJulian: but doesnt init send all procs a KILL?
21:21 semiosis ohhh ok
21:21 semiosis hmmmm
21:22 semiosis never noticed any issue with that
21:22 JoeJulian Otherwise the FIN handshake will never make it to the clients.
21:22 semiosis yeah i understand
21:22 sijis do you still need my logs?
21:22 JoeJulian I've run in to a small handful of people that are having problems with that. I'm not sure why I'm not.
21:22 semiosis right, i never have
21:23 semiosis afaik every time i reboot my gluster servers the clients just disconnect immediately
21:23 semiosis have done many rolling upgrades & never hit a ping timeout afaik
21:23 JoeJulian Me too.
21:23 semiosis (kernel upgrades, not gluster upgrades)
21:24 semiosis sijis: i dont need them at all, but if you have a problem and want me to take a look, i will :)
21:24 xymox joined #gluster
21:29 sprachgenerator joined #gluster
21:33 mattappe_ joined #gluster
21:34 kam270 joined #gluster
21:34 xymox joined #gluster
21:35 sijis semiosis: http://paste.fedoraproject.org/87213/13953513/
21:35 glusterbot Title: #87213 Fedora Project Pastebin (at paste.fedoraproject.org)
21:36 sijis i rebooted around 21:30
21:37 sijis according to /var/log/secure, i executed the command at 21:30:35
21:37 semiosis sijis: this log looks healthy... it shoes intial connection to two replica bricks, then graceful disconnect by the server of one of the bricks, then reconnection to that brick ~20 sec later
21:38 andreask1 joined #gluster
21:38 semiosis sijis: based on this log I would expect the client kept on working, without interruption, with the brick that remained online
21:38 semiosis s/shoes/shows/
21:38 glusterbot What semiosis meant to say was: sijis: this log looks healthy... it shows intial connection to two replica bricks, then graceful disconnect by the server of one of the bricks, then reconnection to that brick ~20 sec later
21:39 sijis semiosis: what's the line htat says it reconnected to node2?
21:39 sijis (just so i know how to look at the log)
21:39 semiosis it starts with this line: [2014-03-20 21:31:23.006723] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0)
21:41 semiosis [2014-03-20 21:31:23.016769] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.1.91.237:49152, attached to remote volume '/export/gv0'.
21:41 semiosis connected.
21:41 sijis i rebooted .237.
21:42 sijis .238 is the other node
21:42 semiosis that is consistent with this log
21:42 xymox joined #gluster
21:43 sijis semiosis: maybe the fact the server takes about a min to get back online after a reboot is skewing my view.
21:44 sijis so regardless, it will 'loose connetivity' for roughly 20 sec
21:44 jag3773 left #gluster
21:44 sijis in that 20 sec window... does it queue whatever its trying to write? (it would seem like it does and i do get 10k files in the brick)
21:49 cjanbanan joined #gluster
21:50 xymox joined #gluster
21:51 sijis semiosis: btw - thanks for looking at the logs
21:51 semiosis yw
21:51 sijis semiosis: did you see my question about that 20 sec window?
21:52 semiosis sijis: it doesnt queue anything... while one brick was disconnected the client continued working with the other brick that stayed online
21:52 kam270 joined #gluster
21:52 semiosis sijis: those files on the "good" brick are marked as having unsynced changes.  when the "bad" brick is back online those changes will be synced to the "bad" brick
21:53 semiosis there should have been no noticeable interruption or delay on the client at all
21:53 sijis semiosis: so the switch is (near) instant?
21:55 semiosis there is no switch
21:55 semiosis the client is continuously connected to all bricks in the volume
21:55 semiosis ,,(mount server)
21:55 glusterbot The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
21:56 semiosis i shouldn't have used the word failover, sorry
21:56 semiosis i should've explained this when you said failover at the beginning
21:56 semiosis the client detects failure, but it doesnt fail "over"
21:57 semiosis it just stops using the failed brick (and starts trying to reconnect)
21:57 rwheeler joined #gluster
21:59 xymox joined #gluster
22:02 sijis semiosis: gotcha.
22:02 sijis this is cool stuff though.
22:03 discretestates joined #gluster
22:07 xymox joined #gluster
22:15 semiosis :D
22:16 xymox joined #gluster
22:17 kam270 joined #gluster
22:19 MugginsM joined #gluster
22:22 nightwalk joined #gluster
22:24 xymox joined #gluster
22:28 refrainblue joined #gluster
22:29 gdubreui joined #gluster
22:31 wrale asking the same question that i did yesterday:  Can anyone imagine why my _gluster_ nfs service would break on one of six mostly identical servers?  i see no sign of any kernel-nfs running.. lsof, netstat and systemd pretty much agree on that.. gluster's nfs.log complains that it can't reach portmap, but it's started (via rpcbind service in F19)..
22:33 xymox joined #gluster
22:34 glusterbot New news from newglusterbugs: [Bug 1021686] refactor AFR module <https://bugzilla.redhat.co​m/show_bug.cgi?id=1021686>
22:37 aixsyd joined #gluster
22:37 aixsyd Heya guys! anyone alive?
22:39 wrale do ignore my previous question.. a: yum erase nfs-utils
22:39 * wrale is dumb
22:40 DV joined #gluster
22:41 wrale funny how nfs-utils is a dependency for glusterfs-regression-tests
22:42 xymox joined #gluster
22:46 MugginsM it's almost as if gluster could do nfs :-P
22:49 xymox joined #gluster
22:51 dtrainor joined #gluster
22:53 jclift wrale: Isn't nfs-utils needed for the Gluster NFS server to work correctly?
22:54 wrale jclift: i'm not sure, honestly.. i'm new to gluster, but  i'm trying very hard to make it work well with ovirt.. i think the ovirt package made some nfs-mobilizing changes that haven't yet revealed themselves to me
22:54 jclift That being said, I had the same nfs service acting up thing on an F19 VM spun up the other day.  Didn't really need it at the time though, so didn't really look into it deeply.
22:54 jclift wrale: Did the oVirt/Gluster guy get back in contact with you?
22:54 jclift (via email)
22:55 jclift Pretty sure I have nfs-utils installed on the CentOS 6.5 VM's that I do testing of Gluster on, if that helps.
22:55 wrale jclift: not to my knowledge, but my box is overflowing with mailing list traffic.. thanks again for passing it on
22:55 recidive joined #gluster
22:56 jclift wrale: Interesting.  Found a guy that said he'd ping you, so I was expecting he would have.
22:56 wrale it must be a service somewhere breaking things.. i need to figure out how to disable nfs once ovirt packages enable it
22:56 jclift Hopefully it's in your Inbox. ;)
22:56 dtrainor Hi.  Having issues mounting a gluster volume.  Sits for a few seconds, then says "Mount failed.  Please check the log file for more detail".  Nothing logs, not on the client nor the server.  Anyone mind helping me out?  http://www.fpaste.org/87235/53561531/
22:56 glusterbot Title: #87235 Fedora Project Pastebin (at www.fpaste.org)
22:57 wrale oh here it is.. lol.. like i said.. buried in mailing list stuff.. thanks.. looking now
22:57 jclift wrale: Do you want to disable nfs on the gluster volumes, or system nfs?
22:57 jclift wrale: There's a gluster volume level option to disable nfs, if that helps.
22:58 wrale jclift: system nfs.. i need gluster nfs for ovirt
22:58 jclift Heh, that option won't help then. :)
22:58 wrale :) right on..
22:58 jclift I'll be quite now, as I'm not really across the bits you're working on.  Good luck, and I hope that guy's email helps. :)
22:58 jclift s/quite/quiet/
22:58 glusterbot What jclift meant to say was: I'll be quiet now, as I'm not really across the bits you're working on.  Good luck, and I hope that guy's email helps. :)
22:58 xymox joined #gluster
22:59 wrale definitely helps.. thank you
23:00 jclift :)
23:00 JoeJulian dtrainor: You missed the client log in that paste.
23:00 dtrainor right, nothing logged, like i said
23:00 dtrainor nothing at all
23:01 JoeJulian /var/log must be full
23:01 dtrainor nope.
23:01 wrale btw, for the world's benefit, a good way to know if system nfs is running while nfs shows as "N" on "gluster volume status": you can run 'rpcinfo -p'.. if nfs is listed there on 2049.. and gluster doesn't seem to be responsible for it, it must be the system's nfs..
23:01 JoeJulian wrale: +1 Good to know, thanks.
23:01 wrale np
23:02 dtrainor the client logs exactly nothing
23:03 kam270 joined #gluster
23:03 JoeJulian does it exist?
23:03 JoeJulian the client log, that is.
23:03 JoeJulian /var/log/glusterfs/net-fileserve​r01.home.lab.dtrainor.local.log
23:03 dtrainor oh, crap
23:04 dtrainor does the log not flush or something?
23:04 dtrainor i was tail -f'ing, nothing cam eup
23:04 JoeJulian tail -F
23:04 dtrainor nice..
23:05 JoeJulian Just a guess.
23:05 dtrainor http://www.fpaste.org/87239/39535668/
23:05 glusterbot Title: #87239 Fedora Project Pastebin (at www.fpaste.org)
23:05 dtrainor i said i tail -f'd
23:05 dtrainor i've noticed tail not tailing logs because, well, whatever is logging doesn't flush
23:05 JoeJulian I don't know why it would have replaced the file, but if it did a lower-case f would follow the old FD and not start a new one.
23:05 dtrainor right
23:06 dtrainor client is fedora20, server is rhel6
23:06 dtrainor [2014-03-20 23:03:52.820082] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-slow_gv00-client-3: DNS resolution failed on host fileserver01.home.lab.dtrainor.local
23:06 dtrainor there's my answer i guess
23:06 dtrainor or, the start of it, at least
23:07 JoeJulian At least it defines your problem a little better. :)
23:07 dtrainor yep.
23:08 xymox joined #gluster
23:08 dtrainor oh, this uncovers a whole nother case of cans of worms
23:08 JoeJulian Hehe
23:08 MugginsM we have a replicated can of worms
23:09 dtrainor indeed.
23:09 JoeJulian At least they're not striped worms.
23:09 MugginsM worms with split brains
23:11 aixsyd jclift: Hey buddy - remember me? Infiniband switch problems?
23:11 wrale_ joined #gluster
23:11 JoeJulian Uh-oh... "hey buddy" is usually a 2 drink minimum.
23:11 aixsyd XD
23:11 aixsyd just secured a Topspin 120 and a Quicksilver 9024 for $300 USD shipped ;)
23:12 dtrainor nailed it.  thanks.
23:12 JoeJulian excellent. You're welcome.
23:13 dtrainor had to... open up my eyes I guess.
23:13 JoeJulian Happens to everyone.
23:15 diegows joined #gluster
23:16 xymox joined #gluster
23:21 andreask joined #gluster
23:23 MugginsM any thoughts on when 3.5 will go release?
23:24 JoeJulian I think I read next week...
23:24 kam270 joined #gluster
23:24 aixsyd any major reasons to upgrade?
23:24 MugginsM we're in a bit of a mess with 3.4.1 but with things messed up in 3.3
23:25 xymox joined #gluster
23:25 MugginsM and building .debs is a pain so I don't want to have to do it with 3.4.2 then 3.4.3 then 3.5 all in a short time :)
23:25 JoeJulian Why do you build your own debs?
23:25 MugginsM Ubuntu Lucid
23:26 JoeJulian Have you built one of the betas?
23:26 MugginsM nope
23:26 MugginsM had to build 3.4 also because we had to patch it to stay on 2400x until we could get a firewall change request through
23:27 JoeJulian I wonder if you'll run across the same dependency deficiency for georeplication as EL5.
23:27 MugginsM new deps?
23:27 MugginsM our servers are precise, just the clients are lucid, is 3.5 server to 3.4 client reasonable?
23:27 JoeJulian I didn't follow it that closely since I only have 2 EL5 boxes left to migrate off of, and I don't use georep.
23:27 MugginsM lucid->trusty upgrade will happen in a few months
23:28 MugginsM also, I'm trying to figure out how to fix split-brain on /
23:28 JoeJulian I don't know how much testing the rpc translation has gotten, but in theory it should be fine.
23:28 MugginsM without erasing an entire side
23:29 JoeJulian Just on "/"? That's just resetting ,,(extended attributes)
23:29 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
23:29 JoeJulian Pick one to set to 0x0.
23:29 MugginsM and repeat for each brick?
23:29 MugginsM (2x5)
23:29 JoeJulian Yes.
23:29 MugginsM 'k, once I've fixed the other problems I'll get to that
23:29 * JoeJulian needs to think of a workaround for split-brain directories...
23:30 MugginsM getting segfaults from self-heal daemon
23:30 JoeJulian eww.
23:30 JoeJulian Have you filed those?
23:30 MugginsM lots of errors in logs
23:30 MugginsM etc
23:30 MugginsM it's not all that well
23:30 MugginsM trying to figure out what I was doing at the time, and if 3.4.2 or 3.4.3 addresses them
23:31 nightwalk joined #gluster
23:32 MugginsM wish I had space/backups to do a fresh setup but I don't :-/
23:32 JoeJulian I hate to say it but I kind-of doubt it.
23:33 JoeJulian I haven't noticed any segfault related bugs in the commit logs for quite a while.
23:33 xymox joined #gluster
23:34 jag3773 joined #gluster
23:36 MugginsM also, servers seem to think there's a 3.3 client out there but I can't find it. At least that's how I read errors from "client-rpc-fops.c:5179:client3_3_inodelk"
23:36 MugginsM would that make sense?
23:38 JoeJulian It's just the rpc call. There hasn't been a new _inodelk since 3.3.
23:38 MugginsM 'k
23:42 jag3773 joined #gluster
23:42 xymox joined #gluster
23:48 elyograg left #gluster
23:49 kam270 joined #gluster
23:51 Elico joined #gluster
23:52 xymox joined #gluster
23:57 coredump joined #gluster
23:59 jrcresawn joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary