Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-11-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 glusterbot New news from newglusterbugs: [Bug 1165429] Gluster Fuse high memory consumption <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165429>
00:10 ttkg joined #gluster
00:11 tryggvil joined #gluster
00:12 topshare joined #gluster
00:28 topshare joined #gluster
00:37 topshare joined #gluster
00:42 topshare joined #gluster
01:06 msmith_ joined #gluster
01:09 glusterbot New news from newglusterbugs: [Bug 1162905] hardcoded gsyncd path causes geo-replication to fail on non-redhat systems <https://bugzilla.redhat.co​m/show_bug.cgi?id=1162905>
01:50 theron joined #gluster
01:54 _Bryan_ joined #gluster
01:59 n-st joined #gluster
02:03 overclk joined #gluster
02:06 haomaiwa_ joined #gluster
02:09 feeshon joined #gluster
02:10 feeshon Hello, does gluster require iptable ports to be opened?
02:24 n-st_ joined #gluster
02:25 n-st_ joined #gluster
02:26 theron joined #gluster
02:31 theron joined #gluster
02:33 unwastable joined #gluster
02:34 kkeithley_ (,,ports)|feeshon
02:34 kkeithley_ @ports
02:34 glusterbot kkeithley_: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up since 3.4.0 (24009 & up previously). (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
02:34 unwastable i have a distributed 2, writing seem very slow (waited 10+ secs to appear)
02:34 kkeithley_ feeshon: ^^^
02:34 unwastable can anybody help?
02:35 unwastable the last msg was:  GlusterFS[15963]: [2014-11-18 22:06:23.639702] C [glusterfsd.c:1224:parse_cmdline] 0-glusterfs: ERROR: parsing the volfile failed (No such file or directory)#012
02:39 side_control joined #gluster
02:41 profit_ joined #gluster
02:42 profit_ hey folks. i'm trying to add a sixth peer to a gluster cluster, and i'm having issues.
02:45 profit_ volume add-brick: failed: Locking failed linux-05. Please check log file for details.
02:45 profit_ is the error i'm getting
02:45 profit_ also i've no clue which log file to look at.
02:45 profit_ which log file am i supposed to be looking at?
02:45 rjoseph joined #gluster
02:45 unwastable testing
02:45 unwastable oh I can't type path in here
02:45 unwastable profit, if it is Ubuntu.. go to /var/log/glusterfs/glusterfsd.log
02:45 unwastable on fedora, centos, redhat will be /var/log/messages
02:47 profit_ http://hastebin.com/ruletogoxa.vbs is what i'm seeing that i think is useful, unwastable
02:47 glusterbot Title: hastebin (at hastebin.com)
02:50 unwastable whats your setup?
02:51 profit_ cent 6, same versions of gluster everywhere (3.6.0)
02:51 profit_ five nodes have the volume added, trying to add a sixth
02:52 unwastable replicated? replicated striping? replicated distributed?
02:52 nishanth joined #gluster
02:52 unwastable or simply distributed
02:53 profit_ replicated
02:53 profit_ just replicated
02:55 unwastable how did they mount? on clients
02:56 profit_ via /etc/fstab: localhost:images /var/lib/one/datastores/1 glusterfs defaults 0 0
02:56 dusmant joined #gluster
03:00 profit_ so yeah no clue why this client doesn't want to add a replica
03:01 harish_ joined #gluster
03:03 unwastable is that all you got in the log?
03:03 profit_ oh, let me actually look at /var/log/messages
03:03 profit_ i was loking at the /var/log/gluster* stuff
03:03 unwastable wont be any
03:04 profit_ i get nothing of use in /var/log/messages
03:04 profit_ and i'm looking at /var/log/glusterfs/glusterfshd.log, right?
03:04 unwastable in that just check /var/log/glusterfs/glusterfsd.log
03:05 profit_ i don't actually have a glusterfsd.log
03:05 unwastable typo
03:05 unwastable thats the one
03:05 profit_ ok, so i'll add in the fshd.log
03:05 unwastable pls do
03:06 profit_ glusterfshd.log doesn't actually show anything
03:06 profit_ when i try the volume add-brick
03:06 profit_ i can kick glusterd and see what happens
03:07 profit_ http://hastebin.com/aliqubabub.vhdl
03:07 glusterbot Title: hastebin (at hastebin.com)
03:09 unwastable no errors..
03:09 profit_ right
03:10 unwastable and after kicking glusterd
03:10 profit_ that was after kicking glusterd
03:10 unwastable how did you add a brick?
03:11 profit_ gluster volume add-brick images replica 6 this-host:/var/lib/one/gfs/images
03:11 unwastable no harm asking have you peer probe the this-host
03:11 profit_ yep
03:12 profit_ all peeers show "in cluster (connected)
03:12 profit_ and volume list shows the 'images' volume
03:15 profit_ i mean i can try blowing away /var/lib/glusterfs
03:17 unwastable unless you havent tried
03:17 profit_ after doing that, it didn't help
03:19 unwastable assuming the data is not live at the moment: destroying the volume and recreate it with a replica 6
03:20 profit_ hrm
03:20 profit_ if i delete and recreate it i'm almost afraid it won't come back
03:20 unwastable you have live data in it?
03:20 profit_ i mean it can be re-created
03:20 profit_ though i've never had issues like this before
03:22 unwastable I dont want to be your google proxy, you probably search better than i do, but please wait up
03:22 profit_ yeah, i might have to just give up and poke this tomorrow
03:24 unwastable wait... is this over VM or real servers?
03:24 profit_ real server
03:25 unwastable please come back here and share your result
03:25 unwastable if you are leaving now
03:25 profit_ yep..
03:25 profit_ left #gluster
03:31 topshare joined #gluster
03:49 unwastable joined #gluster
03:49 unwastable testing
03:50 unwastable i have a distributed 2, creating a file seem very slow (10+ secs to appear), any ideas?
03:52 itisravi joined #gluster
03:55 kanagaraj joined #gluster
03:59 RameshN joined #gluster
04:01 overclk joined #gluster
04:04 sahina_ joined #gluster
04:07 Kins joined #gluster
04:09 kshlm joined #gluster
04:15 Kins joined #gluster
04:15 unwastable can someone help?
04:17 Telsin late night for a lot of people here, have patience
04:17 Telsin what's the network between your nodes?
04:17 unwastable tcp
04:18 unwastable in fact, just one node
04:18 unwastable 2 bricks
04:19 topshare joined #gluster
04:21 Telsin things work in general, this just taking log?
04:21 elyograg joined #gluster
04:21 Telsin *long?
04:21 elyograg I just found a bugzilla describing the NFS crash we have frequently on 3.4.2.
04:21 elyograg bug 1010241
04:21 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1010241 high, unspecified, ---, rgowdapp, CLOSED CURRENTRELEASE, nfs: crash with nfs process
04:22 unwastable the last error was  C [glusterfsd.c:1224:parse_cmdline] 0-glusterfs: ERROR: parsing the volfile failed (No such file or directory)#012
04:22 calisto joined #gluster
04:22 elyograg Upgrading right now isn't an option.  Is this backported to a 3.4 version, or can it be?
04:22 Telsin that was my next questions
04:22 Telsin there is the known issue with local host nfs mounts
04:23 unwastable are you answering my question?
04:23 Telsin new release of 3.4.x just announced, did you check the release notes to see if it addressed it?
04:23 Telsin whups, sorry, mixing responsed to elyograg and you, unwastable
04:24 Telsin unwastable: sorry, don't know that one, running 3.5.2 myself
04:24 unwastable how to upgrade from 3.3 to 3.5.2 without breaking the data
04:25 Telsin the 3.5.x release notes have a method, not sure if you'll need to go through 3.4 or not
04:25 elyograg The patch actually looks pretty simple, but I haven't checked to see if it would apply to 3.4.2 code.  Upgrading to 3.6 isn't something I'd want to try right now.
04:25 Telsin no I wouldn't hit 3.6 yet either, but 3.5 is better for me than 3.4 was
04:26 atalur joined #gluster
04:28 anoopcs joined #gluster
04:28 jiffin joined #gluster
04:29 shubhendu joined #gluster
04:29 nbalachandran joined #gluster
04:48 rafi joined #gluster
04:56 plarsen joined #gluster
04:57 saurabh joined #gluster
04:58 meghanam joined #gluster
04:58 meghanam_ joined #gluster
05:00 unwastable i have a distribute x2 no replication, can i replace a brick with data in it?
05:01 unwastable anyone?
05:01 lalatenduM joined #gluster
05:02 unwastable can someone help?
05:04 plarsen joined #gluster
05:04 atinmu joined #gluster
05:06 hagarth joined #gluster
05:07 gildub joined #gluster
05:10 pp joined #gluster
05:11 topshare joined #gluster
05:15 spandit joined #gluster
05:18 ndarshan joined #gluster
05:24 soumya joined #gluster
05:26 sputnik13 joined #gluster
05:34 ppai joined #gluster
05:35 kshlm joined #gluster
05:35 Telsin unwastable: before it failed? probably. if it failed around, no, you've probably lost some data
05:35 Telsin *allready
05:43 davemc joined #gluster
05:45 soumya joined #gluster
05:50 ramteid joined #gluster
05:50 overclk joined #gluster
06:01 davemc joined #gluster
06:05 nbalachandran joined #gluster
06:06 sputnik13 joined #gluster
06:26 SOLDIERz___ joined #gluster
06:33 nshaikh joined #gluster
06:36 bala joined #gluster
06:47 SOLDIERz joined #gluster
06:48 LebedevRI joined #gluster
06:49 soumya joined #gluster
06:51 kumar joined #gluster
06:52 nbalachandran joined #gluster
07:02 dusmant joined #gluster
07:06 dusmant joined #gluster
07:08 glusterbot New news from resolvedglusterbugs: [Bug 1051896] setfacl with "invalid argument" will return 0, not return EINVAL in glusterfs <https://bugzilla.redhat.co​m/show_bug.cgi?id=1051896>
07:12 ricky-ticky joined #gluster
07:18 vimal joined #gluster
07:33 dusmantkp_ joined #gluster
07:40 glusterbot New news from newglusterbugs: [Bug 1165545] Program terminated with signal 11, Segmentation fault.glusterfs_this_location@plt <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165545>
07:52 Fen2 joined #gluster
08:08 mbukatov joined #gluster
08:14 ricky-ticky joined #gluster
08:14 flu_ joined #gluster
08:15 flu_ hi
08:15 glusterbot flu_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
08:15 flu_ today I saw a strange issue.
08:15 flu_ When I use dd to copy file from local disk to glusterfs volume
08:15 flu_ the first 4KB is all zero
08:18 flu_ other bytes of the files in glusterfs are the same to the local file
08:19 Debloper joined #gluster
08:19 flu_ But when I try again, the two files are the same
08:19 flu_ https://drive.google.com/file/d/0ByKuS8​1H5x1VRDV2R291V0padlU/view?usp=sharing
08:19 glusterbot Title: error.png - Google Drive (at drive.google.com)
08:20 flu_ you could got some from the image
08:37 afics_ joined #gluster
08:38 ricky-ticky1 joined #gluster
08:40 SOLDIERz joined #gluster
08:41 fsimonce joined #gluster
08:42 elico joined #gluster
08:44 soumya_ joined #gluster
08:46 T0aD joined #gluster
08:47 karnan joined #gluster
08:48 nbalachandran joined #gluster
08:49 dusmantkp__ joined #gluster
09:00 dusmant joined #gluster
09:01 bala joined #gluster
09:08 LebedevRI joined #gluster
09:11 deepakcs joined #gluster
09:11 tryggvil joined #gluster
09:13 suliba joined #gluster
09:13 jiffin joined #gluster
09:23 Fen1 joined #gluster
09:24 Debloper joined #gluster
09:26 elico joined #gluster
09:28 bernardo joined #gluster
09:29 bernardo Hi
09:29 glusterbot bernardo: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:31 capri joined #gluster
09:36 nshaikh joined #gluster
09:37 MrAbaddon joined #gluster
09:38 Philambdo joined #gluster
09:45 ppai joined #gluster
09:52 hagarth joined #gluster
09:54 rjoseph joined #gluster
09:57 DV joined #gluster
10:14 deepakcs joined #gluster
10:27 soumya_ joined #gluster
10:31 hagarth joined #gluster
10:43 shubhendu joined #gluster
10:45 tvb joined #gluster
10:46 tvb Guys who can assist me on how to verify if glusterfs locking mechanism is enabled?
10:47 ndarshan joined #gluster
10:47 nbalachandran joined #gluster
10:50 afics joined #gluster
10:56 tvb Anyone?
11:01 [Enrico] joined #gluster
11:12 shubhendu joined #gluster
11:12 ndarshan joined #gluster
11:13 harish_ joined #gluster
11:15 ira joined #gluster
11:21 diegows joined #gluster
11:34 mator http://gluster.org/community/documentation/index.p​hp/GlusterFS_Technical_FAQ#How_is_locking_handled.3F
11:34 calisto joined #gluster
11:36 mator i wonder what the problem to take 2 fs clients and check locking "google >> fs file locking sample, linux"
11:36 ctria joined #gluster
11:37 haomaiwa_ joined #gluster
11:38 aravinda_ joined #gluster
11:40 davemc Gluster community meeting in 20 minutes, on #gluster-meeting
11:45 dusmant joined #gluster
11:47 meghanam__ joined #gluster
11:50 meghanam joined #gluster
11:56 jmarley joined #gluster
11:56 jmarley_ joined #gluster
12:01 soumya_ joined #gluster
12:05 hagarth joined #gluster
12:10 SOLDIERz joined #gluster
12:15 jdarcy joined #gluster
12:15 bala joined #gluster
12:16 itisravi_ joined #gluster
12:23 chirino joined #gluster
12:35 tdasilva joined #gluster
12:38 tvb joined #gluster
12:44 edward1 joined #gluster
12:51 ctria joined #gluster
12:58 Fen1 joined #gluster
13:01 bala1 joined #gluster
13:01 bennyturns joined #gluster
13:11 ctria joined #gluster
13:13 ctrianta joined #gluster
13:16 aravinda_ joined #gluster
13:23 tryggvil joined #gluster
13:33 bene2 joined #gluster
13:49 nbalachandran joined #gluster
13:52 nishanth joined #gluster
13:52 anoopcs joined #gluster
13:57 theron joined #gluster
13:58 theron joined #gluster
14:00 skippy we continue to get weird FUSE client disconnects with Gluster 3.5.2.  Last night we had a Gluster "ping timeout", but it was then resolved within the same second: https://gist.github.com/skpy/55bdfe6cc7e2fc7fb0ee
14:00 skippy we're not seeing any real pattern for when these ping timeouts occur.  Sometimes at night, sometimes during the work day.
14:00 tvb left #gluster
14:00 skippy Sometimes they resolve in under a second, sometimes it takes upwards of 10 seconds to resolve.
14:01 bernardo i have the same problem :(
14:04 rwheeler joined #gluster
14:04 meghanam__ joined #gluster
14:05 meghanam joined #gluster
14:07 Slashman joined #gluster
14:08 topshare joined #gluster
14:09 topshare joined #gluster
14:12 virusuy_ joined #gluster
14:12 topshare joined #gluster
14:17 bala joined #gluster
14:17 dusmant joined #gluster
14:18 bernardo ping timeout on the first server : http://pastebin.com/raw.php?i=h4KidFde ...then on the backup server : http://pastebin.com/raw.php?i=pQqBzTfQ
14:18 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
14:18 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
14:20 rjoseph joined #gluster
14:21 coredump joined #gluster
14:24 bernardo skippy: what os/kernel are you using ?
14:24 stickyboy joined #gluster
14:24 msmith joined #gluster
14:25 FrankLu joined #gluster
14:25 msmith joined #gluster
14:28 topshare joined #gluster
14:28 shubhendu joined #gluster
14:37 skippy bernardo: client is RHEL6.5 2.6.32-431.20.3.el6.x86_64
14:37 skippy servers are RHEL7
14:38 skippy my servers never time out between each other.  It's just the client communicating to the servers.
14:39 meghanam__ joined #gluster
14:40 meghanam joined #gluster
14:40 tryggvil joined #gluster
14:40 skippy at the same timestamp as my client reports the ping timeout, my servers report nothing unusual, so far as I can tell.
14:41 stickyboy joined #gluster
14:41 skippy 15 seconds BEFORE the timeout, one server reports "[2014-11-19 02:15:04.339037] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: bb61111b-b048-4dc9-b54d-12a0cc2dd8a9, lock held by: 23989211-4f0d-4087-
14:41 skippy b9c5-bc82295b2c38
14:41 skippy [2014-11-19 02:15:04.339053] E [glusterd-syncop.c:1221:gd_sync_task_begin] 0-management: Un
14:41 skippy able to acquire lock"
14:43 skippy servers are on one subnet, clients on another.  Not seeing anything in switch, router, or firewall logs to indicate network hiccups of any kind.
14:45 _Bryan_ joined #gluster
14:45 bernardo weird...i have errors on both client/server sides (first messages on client: blah has not responded )
14:46 bernardo are you sure the servers are logging correctly ?
14:47 bernardo i had some strange problems where i had to restart gluster server processes, because there was no more logs
14:47 bernardo logging*
14:48 bernardo i have client/server on the same nodes (debian/proxmox distrib with centos 7 kernel)
14:48 plarsen joined #gluster
14:48 skippy i'm not seeing any server-side logs for this brick, but I am seeing logs in /var/log/gluster/etc-glusterfs-glusterd.vol.log
14:50 bernardo maybe i am wrong, but i think you should see clients connecting in the logs, like this :
14:50 bernardo [2014-11-16 21:13:31.162496] I [client_t.c:417:gf_client_unref] 0-glusterstorage-server: Shutting down connection proxmoxt2-13212-2014/11/06-09:54:34:​958311-glusterstorage-client-16-0-0
14:50 bernardo [2014-11-16 21:13:36.178309] I [server-handshake.c:575:server_setvolume] 0-glusterstorage-server: accepted client from proxmoxt2-13212-2014/11/06-09:54:34:​958311-glusterstorage-client-16-0-1 (version: 3.5.2)
14:50 skippy in the brick log?
14:50 RameshN joined #gluster
14:50 bernardo yes
14:51 skippy hrm
14:52 plarsen joined #gluster
14:52 calisto1 joined #gluster
14:55 bala joined #gluster
14:58 kanagaraj joined #gluster
14:59 bene joined #gluster
15:03 plarsen joined #gluster
15:04 skippy bizarro.  I unmounted the volume on the client,  then remounted it.  No entries in the brick log.
15:04 skippy But a coworker built a new volume this morning, and I do see log entries in that volume's brick log.
15:07 bernardo i think i was misleading, replicated bricks are subvolumes named 'clients' in the bricks logs.. :s
15:07 rjoseph joined #gluster
15:10 skippy so the individual clients (that mount the volume) would not be logging to the brick log?
15:13 ghenry joined #gluster
15:14 wushudoin joined #gluster
15:15 Fen1 joined #gluster
15:20 sage_ joined #gluster
15:22 soumya_ joined #gluster
15:26 msmith_ joined #gluster
15:28 bernardo it make sense.. :/
15:30 bernardo gluster logs are a bit cryptic to me
15:40 sage_ joined #gluster
15:42 tryggvil joined #gluster
15:49 tdasilva joined #gluster
15:49 kshlm joined #gluster
15:54 bala joined #gluster
15:54 msmith_ joined #gluster
15:57 mojibake joined #gluster
16:01 meghanam joined #gluster
16:01 lpabon joined #gluster
16:02 meghanam__ joined #gluster
16:03 bala joined #gluster
16:12 skroz joined #gluster
16:12 jmarley_ joined #gluster
16:16 jobewan joined #gluster
16:19 bene joined #gluster
16:30 toti joined #gluster
16:35 lalatenduM joined #gluster
16:42 smohan joined #gluster
16:44 smohan_ joined #gluster
16:46 hagarth joined #gluster
16:47 kumar joined #gluster
16:49 Fen1 joined #gluster
16:49 bernardo Is it normal to have more than 9000 occurences of "/var/lib/glusterfs/sdc1/.gluste​rfs/indices/base_indices_holder" in /proc/<glusterfsd PID>/fd ?
16:53 rjoseph joined #gluster
16:53 MrAbaddon joined #gluster
16:54 lmickh joined #gluster
16:58 bernardo node1 file-nr :888384 node2 file-nr :9312 node3 file-nr :9408
17:02 sputnik13 joined #gluster
17:02 RameshN joined #gluster
17:06 PeterA joined #gluster
17:08 ildefonso joined #gluster
17:08 maveric_amitc_ joined #gluster
17:09 elico joined #gluster
17:16 nbalachandran joined #gluster
17:17 bernardo Here are the open fd on each bricks : http://fpaste.org/152211/16417277/raw/
17:18 _dist joined #gluster
17:18 anoopcs joined #gluster
17:24 calisto joined #gluster
17:24 anoopcs joined #gluster
17:29 anoopcs joined #gluster
17:37 virusuy joined #gluster
17:51 sputnik13 joined #gluster
17:52 msmith__ joined #gluster
17:57 maveric_amitc_ joined #gluster
18:03 nshaikh joined #gluster
18:04 theron joined #gluster
18:06 nishanth joined #gluster
18:08 msmith_ joined #gluster
18:18 maveric_amitc_ joined #gluster
18:23 sputnik13 joined #gluster
18:38 RameshN joined #gluster
18:39 ildefonso joined #gluster
18:45 MugginsM joined #gluster
18:46 tdasilva joined #gluster
18:48 profit_ joined #gluster
18:48 profit_ so i'm having a bad time with my cluster
18:49 profit_ i can't add a new brick to a volume (replicated) and two boxes are showing as offline.
18:49 profit_ what's the best way to try to 1.) get these nodes showing the volume as online, and 2.) get the last brick added?
18:51 Paul-C joined #gluster
18:51 semiosis profit_: ,,(pasteinfo)
18:51 semiosis glusterbot: ping
18:52 glusterbot profit_: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
18:52 glusterbot pong
18:52 profit_ sure
18:52 semiosis glusterbot: whoami
18:52 glusterbot semiosis: I don't recognize you.
18:53 profit_ semiosis: https://dpaste.de/iwM6
18:53 glusterbot Title: dpaste.de: Snippet #291999 (at dpaste.de)
18:54 semiosis profit_: replica 5? really?
18:54 semiosis which of these volumes are you having trouble with?
18:54 profit_ images
18:54 profit_ i'm just trying to replicate to every node...
18:54 profit_ also https://dpaste.de/fmaa is volume status
18:54 glusterbot Title: dpaste.de: Snippet #292000 (at dpaste.de)
18:56 semiosis you have two bricks down.  go to those servers (sse-on & sse-on-03) and restart glusterd, then if the bricks are still not started, check the brick log file for more info
18:56 profit_ sure
18:56 semiosis actually those servers show all bricks down
18:56 semiosis could be a host issue
18:56 semiosis hard to say
18:57 profit_ after bouncing, looks like nothing has changed.
18:58 profit_ do i have to do a volume repair or something?
18:59 profit_ i'm also seeing [2014-11-19 18:56:37.030369] E [posix.c:5604:init] 0-images-posix: Extended attribute trusted.glusterfs.volume-id is absent
18:59 profit_ in the brick log
18:59 semiosis a ha.  is /var/lib/one/gfs/ mounted?
18:59 nshaikh left #gluster
18:59 semiosis looks like your brick dir either is not mounted or got wiped
18:59 profit_ uhh
19:00 profit_ /var/lib/one/gfs has files in it, yes.
19:00 semiosis somehow the xattr is missing
19:00 profit_ though it's not mounted via gluster
19:01 semiosis when you created the volume (or added that brick) glusterfs wrote an xattr to the brick dir.  that's missing now, which means either that device is no longer mounted, or the directory got deleted.
19:01 profit_ party
19:01 semiosis see ,,(extended attributes)
19:01 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
19:01 profit_ is there a way to tell gluster to re-create those xattrs?
19:01 semiosis you can look at the xattrs of the other brick dirs & compare
19:02 semiosis glusterbot: you laggin
19:02 semiosis glusterbot: reconnect
19:02 glusterbot semiosis: Error: You don't have the owner capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
19:02 semiosis glusterbot: meh
19:02 glusterbot semiosis: I'm not happy about it either
19:03 profit_ yeah, the xattrs aren't there.
19:04 profit_ so do i get to go in with a blowtorch and setfattr?
19:04 semiosis i'd use rsync -aHAX --whole-file --inplace to sync from one of the other replicas
19:04 profit_ sure
19:04 maveric_amitc_ joined #gluster
19:04 semiosis then it should start right up
19:04 profit_ i'm assuming stopping glusterd first would be a Good Call ?
19:05 semiosis well the brick is already dead so it shouldnt matter
19:05 diegows joined #gluster
19:05 semiosis see ,,(processes)
19:05 glusterbot The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal).
19:05 semiosis the brick export daemon for that directory is not running, so it shouldnt matter that you're mucking around in there
19:07 semiosis also, fwiw, having more than 3 replicas is usually an antipattern.
19:07 semiosis maybe you dont really need to have that data replicated to a 6th machine
19:07 semiosis just put a client mount on it
19:07 semiosis and read from the others
19:08 semiosis i suppose there are odd cases where so many replicas makes sense, so maybe
19:08 profit_ sure
19:08 profit_ well it fixed it on one box so let's try the other
19:09 semiosis woo!
19:09 semiosis lunchtime
19:14 profit_ alright, trying to add the last node in leaves us with "volume add-brick: failed: One or more nodes do not support the required op-version. Cluster op-version must atleast be 30600."
19:29 MrAbaddon joined #gluster
19:38 Slydder joined #gluster
19:38 Slydder hey all
19:40 Slydder So. After running Gluster for a week mounted using NFS (Ganesha) I had a massive deadlock which forced me to go back to gluster fuse mounts. This is just crazy how slow this is.
19:40 Slydder I heard there is a way to shut off the heal call on lookups. is this true and if so HOW?!?!?!?!
19:42 Slydder Anything to get this to an acceptable level of performance. With the use of cachefilesd I was getting acceptable speeds for our Magento shops but with the current state of NFS in Linux it's just not viable.
19:42 tryggvil joined #gluster
19:47 Slydder anyone?
19:52 semiosis ,,(options)
19:52 glusterbot See config options and their defaults with 'gluster volume set help'; you can see the current value of an option, if it has been modified, with 'gluster volume info'; see also this page about undocumented options: http://goo.gl/mIAe4E
19:52 semiosis that link might have info on disabling the self heal checks ^^
19:53 semiosis Slydder: interesting to see that even ganesha isn't immune to the localhost nfs deadlock
19:54 semiosis profit_: glusterfs versions different?
19:54 profit_ semiosis: they're all the same version
20:03 profit_ semiosis: glusterd.info shows operating-version=1
20:06 ricky-ticky joined #gluster
20:13 tdasilva joined #gluster
20:33 n-st joined #gluster
20:35 tom[] is it preferable for robustness to have at least 3 nodes in a replication gluster? for example, is there a split-brain hazzard in a 2-node gluster?
20:38 julim joined #gluster
20:38 kkeithley there's a split-brain hazard in a 3-node gluster too.  The reason to use three nodes though, is to provide quorum. With three nodes gluster has an easy way to determine which is the good copy
20:39 kkeithley s/good/bad/
20:39 glusterbot What kkeithley meant to say was: there's a split-brain hazard in a 3-node gluster too.  The reason to use three nodes though, is to provide quorum. With three nodes gluster has an easy way to determine which is the bad copy
20:39 tom[] yikes, that bot tries too hard
20:40 tom[] kkeithley: ok. that makes sense
20:40 kkeithley it's a very good bot
20:40 tom[] tnx
20:40 semiosis tom[]++
20:40 glusterbot semiosis: tom [] 's karma is now 1
20:41 tom[] semiosis: i have no idea what that means but thanks anyway
20:41 julim joined #gluster
20:42 semiosis tom[]: ,,(thx)
20:42 glusterbot tom [] : you're welcome
20:42 tom[] is the bot provided by the logging service?
20:42 ctria joined #gluster
20:42 semiosis !m tom[]
20:42 [o__o] You're doing good work, tom[]!
20:42 semiosis that's the logging bot
20:42 semiosis glusterbot is our own thing
20:43 tom[] is it open source?
20:43 semiosis indeed
20:43 semiosis @version
20:43 glusterbot semiosis: The current (running) version of this Supybot is 0.83.4.1+limnoria 2014.03.25, running on Python 2.6.6 (r266:84292, Jul 10 2013, 22:48:45)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]. The newest versions available online are 2014.11.17 (in testing), 2014.10.09 (in master).
20:44 JoeJulian hmm, I should update that.
20:44 semiosis JoeJulian++
20:44 glusterbot semiosis: JoeJulian's karma is now 15
20:44 JoeJulian ... probably on a cron schedule...
20:44 semiosis give yourself some more karma while you're in there too
20:44 tom[] JoeJulian: glusterbot should have updated that already ;)
20:44 semiosis you've earned it
20:44 JoeJulian I like to blend in.
20:44 JoeJulian I'm a karma chameleon.
20:44 semiosis wow
20:45 * tom[] sings
20:45 kkeithley scary
20:48 tom[] huh? i have a fine contralto
20:51 kkeithley no, I meant, JoeJulian being a karma chameleon is scary
20:51 skippy setting "priority=50" on the upstream gluster yum repo does not seem to allow me to install a specific older version of Gluster from that repo:  yum keeps trying to update glusterfs-libs from the RHEL channel :(
20:51 zerick joined #gluster
20:52 kkeithley skippy: you need to also install the yum-plugin-priorities RPM for that to work
20:52 skippy d'oh!
20:52 skippy thanks!
20:53 kkeithley yw
20:53 kkeithley @yum
20:53 glusterbot kkeithley: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
20:53 coredump joined #gluster
20:54 kkeithley @rhel
20:54 glusterbot kkeithley: I do not know about 'rhel', but I do know about these similar topics: 'rhel6 deadlocks'
20:54 kkeithley @forget yum
20:54 glusterbot kkeithley: The operation succeeded.
20:55 kkeithley @learn yum as The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 19 and later are in the Fedora yum updates (or updates-testing) repository.
20:55 glusterbot kkeithley: The operation succeeded.
20:57 ultrabizweb joined #gluster
20:57 kkeithley @repo
20:57 glusterbot kkeithley: I do not know about 'repo', but I do know about these similar topics: 'git repo', 'ppa repo', 'repos', 'repository', 'yum repo'
20:58 kkeithley @repos
20:58 glusterbot kkeithley: See @yum, @ppa or @git repo
20:58 kkeithley @repository
20:58 glusterbot kkeithley: git clone https://github.com/gluster/glusterfs.git
20:58 kkeithley @yum repo
20:58 glusterbot kkeithley: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates (or updates-testing) repository.
20:58 kkeithley @forget yum repo
20:58 glusterbot kkeithley: The operation succeeded.
20:58 kkeithley @learn yum repo as The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 19 and later are in the Fedora yum updates (or updates-testing) repository.
20:58 glusterbot kkeithley: The operation succeeded.
20:58 kkeithley @yum
20:58 glusterbot kkeithley: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 19 and later are in the Fedora yum updates (or updates-testing) repository.
20:58 kkeithley @yum repo
20:58 glusterbot kkeithley: The official community glusterfs packages for RHEL (including CentOS, SL, etc) are available at http://download.gluster.org/pub/gluster/glusterfs/. The official community glusterfs packages for Fedora 19 and later are in the Fedora yum updates (or updates-testing) repository.
20:59 skippy upgraded to 3.6.1 today on servers and client.  FUSE client still showing random intermittent ping timeouts: https://gist.githubusercontent.com/skpy/c​927399e30c9a61c3b27/raw/b49a20956c6417488​902bdb7392fd1cdef9b4a52/ping-timeout.md
20:59 skippy this is most distressing.
21:00 semiosis skippy: every single time i've seen someone report random intermittent ping timeouts it has turned out to be a networking issue
21:00 skippy i don't doubt it; but we're not seeing anything in our switch, router, or firewall logs to indicate any kind of problem.
21:00 semiosis most of the time people insist it can't *possibly* be a networking issue... then they figure out that it is
21:00 skippy the client reports the error; the servers never complain.
21:00 semiosis trunking?
21:01 semiosis ping timeout is a client thing
21:01 skippy ah, good to know I can stop looking for server-server errors at the same time.
21:02 profit_ semiosis: is there a way to fix volume add-brick: failed: One or more nodes do not support the required op-version. Cluster op-version must atleast be 30600. sort of errors?
21:02 skippy VMware clint going to physical servers.  replica 2.  client on one subnet, servers on another.  both servers on same subnet.
21:02 semiosis profit_: well i tried to discourage you from adding a 6th replica....
21:02 profit_ yeah..
21:02 JoeJulian 6?!?!?!
21:03 profit_ the thought would be every host has a copy of the data.
21:03 profit_ local copy, even.
21:03 JoeJulian profit_: The servers all have to be running the same version.
21:03 semiosis JoeJulian: i knew you'd like that
21:03 profit_ oddly enough they are all running the same version of the gluster packages
21:03 JoeJulian nope
21:03 JoeJulian They may report that they all have the same version installed, but they're not all running the same version.
21:04 profit_ cool
21:04 kkeithley learn yum priorities as To prefer community repos over the distribution's, e.g. on RHEL/CentOS 6.6 or 7.0, add "priority=50" to your /etc/yum.repos.d/glusterfs-epel.repo and install the yum-plugin-priorities RPM. (And check your other repos for a priority= setting!)
21:04 JoeJulian Check "lsof | grep deleted"
21:04 profit_ JoeJulian: this is persisting after even rebooting
21:04 kkeithley @learn yum priorities as To prefer community repos over the distribution's, e.g. on RHEL/CentOS 6.6 or 7.0, add "priority=50" to your /etc/yum.repos.d/glusterfs-epel.repo and install the yum-plugin-priorities RPM. (And check your other repos for a priority= setting!)
21:04 glusterbot kkeithley: The operation succeeded.
21:04 JoeJulian profit_: You rebooted all your servers?
21:05 profit_ i did
21:05 profit_ all running 3.6.1-1
21:07 julim joined #gluster
21:10 JoeJulian profit_: check all your gluster logs for "operating version .* is more than the maximum supported (.*) on the machine (as per peer request from"
21:19 glusterbot joined #gluster
21:20 JoeJulian @version
21:20 glusterbot JoeJulian: The current (running) version of this Supybot is 0.83.4.1+limnoria 2014.10.09, running on Python 2.6.6 (r266:84292, Jul 10 2013, 22:48:45)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)].  The newest versions available online are 2014.11.17 (in testing), 2014.10.09 (in master).
21:28 profit_ JoeJulian: i don't really see that, it looks like I can do gluster volume set all cluster.op-version
21:42 profit_ .. which fixed it.
21:42 JoeJulian Nice! Thanks for that, I didn't know that was a problem.
21:43 profit_ yeah
21:43 profit_ i just set the op-verion to 30600 and it STFU'd
21:46 mrint21h joined #gluster
21:46 mrint21h Hi all
21:47 JoeJulian @mp add "support the required op-version" "echo $nick: set the desired op-version using ''gluster volume set all cluster.op-version $desired_op_version''."
21:47 glusterbot JoeJulian: The operation succeeded.
21:47 mrint21h I have over 10TB in a gluster arrangement and gluster just does not work anymore. I cannot peer probe, volume info nothing
21:47 mrint21h 3.3. GlusterFS
21:48 mrint21h I need to fix some problems with some nodes
21:48 semiosis mrint21h: maybe if you could pastie.org a log file, or something
21:48 profit_ glusterbot: are you skynet?
21:48 tryggvil joined #gluster
21:49 glusterbot News from newglusterbugs: [Bug 1163623] Erasure Volume quota error <https://bugzilla.redhat.co​m/show_bug.cgi?id=1163623>
21:49 glusterbot News from newglusterbugs: [Bug 1165010] Regression TestFrameWork : Starting a process fails with "Port already in use" error in our regression test framework <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165010>
21:49 glusterbot News from newglusterbugs: [Bug 1165429] Gluster Fuse high memory consumption <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165429>
21:49 glusterbot News from newglusterbugs: [Bug 1165545] Program terminated with signal 11, Segmentation fault.glusterfs_this_location@plt <https://bugzilla.redhat.co​m/show_bug.cgi?id=1165545>
21:49 profit_ mrint21h: gluster volume status says?
21:49 glusterbot News from newglusterbugs: [Bug 1126831] Memory leak in GlusterFs client <https://bugzilla.redhat.co​m/show_bug.cgi?id=1126831>
21:49 glusterbot News from resolvedglusterbugs: [Bug 1051896] setfacl with "invalid argument" will return 0, not return EINVAL in glusterfs <https://bugzilla.redhat.co​m/show_bug.cgi?id=1051896>
21:49 profit_ also were any of you folks at LISA last week?
21:49 glusterbot profit_: all your base are belong to us
21:50 profit_ @grab mrint21h
21:50 profit_ oh ok this isn't super goofy supybot
21:52 profit_ left #gluster
21:52 badone joined #gluster
21:52 mrint21h http://pastie.org/9730981
21:52 mrint21h gluster volume returns nothing
21:55 mrint21h @profit_t  I am not sure if I can recompile gluster only, is st something wrong with it? The filesystem is mounted under /mnt/glusterfs
21:57 JoeJulian @later tell profit_ Yes, I was at LISA.
21:57 glusterbot JoeJulian: The operation succeeded.
22:11 badone joined #gluster
22:35 elyograg red gold and green.
23:23 MugginsM joined #gluster
23:28 gildub joined #gluster
23:40 nage joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary