Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-25

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 jag3773 joined #gluster
00:18 divbell joined #gluster
00:19 marcoceppi joined #gluster
00:19 jobewan joined #gluster
00:28 gdubreui joined #gluster
00:29 jobewan joined #gluster
00:31 Philambdo joined #gluster
00:32 jobewan joined #gluster
00:34 jobewan joined #gluster
00:36 jobewan joined #gluster
00:37 jobewan joined #gluster
00:47 jporterfield joined #gluster
00:48 jobewan joined #gluster
00:49 vpshastry joined #gluster
00:53 Alex joined #gluster
00:56 elyograg joined #gluster
00:56 elyograg does anything need to be changed as far as config goes to enable the readdirp support in gluster as long as the OS supports it?
00:57 elyograg i'm upgrading to 3.4.2 now, want to be sure it will work, and if not, find out what i need to do.
01:03 cjanbanan joined #gluster
01:09 tokik joined #gluster
01:13 khushildep joined #gluster
01:25 diegows joined #gluster
01:31 cjanbanan joined #gluster
01:56 elyograg my second volume shows some constant CPU usage on the brick servers, and any attempt to get heal info on that volume doesn't work - it spits nothing out for a long time, then just exits with no output.  Restarting glusterd is required to wake everything up and allow other volume commands to run.
01:59 kevein joined #gluster
02:00 jmarley joined #gluster
02:00 jmarley joined #gluster
02:01 cjanbanan joined #gluster
02:12 elyograg why do all the logs say something like "Program: GlusterFS 3.3" when it's 3.4.2?
02:32 sprachgenerator joined #gluster
02:45 jag3773 joined #gluster
02:55 ColPanik joined #gluster
03:01 nightwalk joined #gluster
03:01 cjanbanan joined #gluster
03:06 ColPanik joined #gluster
03:20 cp0k_ joined #gluster
03:28 dusmant joined #gluster
03:32 cjanbanan joined #gluster
03:35 aquagreen joined #gluster
03:39 shubhendu joined #gluster
03:43 itisravi joined #gluster
03:47 kanagaraj joined #gluster
03:50 nueces joined #gluster
03:51 sac`away joined #gluster
03:57 aquagreen joined #gluster
03:58 ndarshan joined #gluster
04:00 20WABCJ3V joined #gluster
04:07 RameshN joined #gluster
04:08 sahina joined #gluster
04:19 saurabh joined #gluster
04:19 shylesh joined #gluster
04:20 ppai joined #gluster
04:25 cp0k__ joined #gluster
04:32 cjanbanan joined #gluster
04:41 sputnik13 joined #gluster
04:42 jag3773 joined #gluster
04:47 sputnik13 joined #gluster
04:52 ira joined #gluster
04:56 spandit joined #gluster
04:57 bala joined #gluster
05:00 prasanth joined #gluster
05:02 cjanbanan joined #gluster
05:02 nueces joined #gluster
05:06 tokik joined #gluster
05:06 hagarth joined #gluster
05:10 kdhananjay joined #gluster
05:21 kanagaraj joined #gluster
05:22 vpshastry joined #gluster
05:23 ndarshan joined #gluster
05:25 bala1 joined #gluster
05:28 rastar joined #gluster
05:28 RameshN joined #gluster
05:30 aquagreen joined #gluster
05:34 raghu joined #gluster
05:55 rastar joined #gluster
05:59 benjamin_____ joined #gluster
05:59 sputnik13 purpleidea: you around?
06:02 harish joined #gluster
06:02 cjanbanan joined #gluster
06:30 nshaikh joined #gluster
06:33 jporterfield joined #gluster
06:36 Philambdo joined #gluster
06:37 satheesh joined #gluster
06:46 jporterfield joined #gluster
06:48 vimal joined #gluster
06:54 wushudoin joined #gluster
07:01 rastar joined #gluster
07:01 cjanbanan joined #gluster
07:16 glusterbot New news from newglusterbugs: [Bug 1069494] DHT - In rebalance(after add-brick or sub-vol per dir change) hash layout is not re distributed properly. It can be optimized to reduce file migration <https://bugzilla.redhat.com/show_bug.cgi?id=1069494>
07:23 jtux joined #gluster
07:24 tjikkun_work joined #gluster
07:25 TvL2386 joined #gluster
07:30 shubhendu joined #gluster
07:32 hagarth joined #gluster
07:34 sputnik13 err...  help? :)
07:34 sputnik13 tried to add a volume and it failed...  tried to create again and it complains about the brick already being part of a volume
07:34 sputnik13 but when I do list volume nothing shows
07:36 elyograg what does it mean when I have "N/A" for the brick port number on gluster volume status?
07:36 elyograg http://fpaste.org/80106/93313668/
07:36 glusterbot Title: #80106 Fedora Project Pastebin (at fpaste.org)
07:42 elyograg restarting glusterd didn't help, but stopping glusterd, stopping glusterfsd, then killing all glusterfs processes, then restarting glusterd *did* fix it.
07:45 rgustafs joined #gluster
07:48 ekuric joined #gluster
07:50 bala1 joined #gluster
07:50 rossi_ joined #gluster
08:02 eseyman joined #gluster
08:10 hagarth joined #gluster
08:14 keytab joined #gluster
08:14 kanagaraj joined #gluster
08:14 ndarshan joined #gluster
08:15 Rydekull joined #gluster
08:15 tjikkun_work joined #gluster
08:17 ctria joined #gluster
08:21 cjanbanan joined #gluster
08:23 sahina joined #gluster
08:24 bala1 joined #gluster
08:24 rgustafs joined #gluster
08:24 andreask joined #gluster
08:28 elyograg joined #gluster
08:30 aravindavk joined #gluster
08:33 ProT-0-TypE joined #gluster
08:39 prasanth joined #gluster
08:44 fsimonce joined #gluster
08:45 vincent_vdk joined #gluster
08:53 bala1 joined #gluster
08:53 fsimonce joined #gluster
08:55 RameshN joined #gluster
08:58 Copez joined #gluster
08:58 Copez Question about brick-sizing..
08:59 Copez I have a Gluster of 12 TB, which will be used for KVM
09:00 Copez Should I make one brick of 12 TB or i.e. 20x 600GB
09:03 liquidat joined #gluster
09:07 satheesh joined #gluster
09:08 wica joined #gluster
09:09 wica Hi, is there a howto on how to replace a failed disk on gluster 3.3.2 ?
09:29 jporterfield joined #gluster
09:58 badone joined #gluster
10:04 lalatenduM joined #gluster
10:05 ngoswami joined #gluster
10:20 bala1 joined #gluster
10:24 jmarley joined #gluster
10:24 jmarley joined #gluster
10:29 shyam joined #gluster
10:31 spandit joined #gluster
10:32 sulky joined #gluster
10:36 kdhananjay joined #gluster
10:39 davinder joined #gluster
10:42 NuxRo ndevos: will test the gluster ACS patches in 4.3 later today, will report my findings
10:42 borreman_123 joined #gluster
10:45 hybrid512 joined #gluster
10:47 rjoseph joined #gluster
10:47 X3NQ joined #gluster
10:48 ctria joined #gluster
10:49 sprachgenerator joined #gluster
10:53 mbukatov joined #gluster
10:57 prasanth joined #gluster
11:08 khushildep joined #gluster
11:14 satheesh joined #gluster
11:20 ndevos NuxRo: many thanks!
11:21 ndevos NuxRo: also, all patches have now been merged in CloudStack/master, 4.4 shoud support Gluster out of the box
11:21 abyss^ I add 4 glusterfs servers to peers but first (where I do gluster peer probe) have ip instead of hostname why? Can I manually change it in /var/lib/glusterd/peers/ files?
11:21 ndevos ~hostnames | abyss^
11:21 glusterbot abyss^: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
11:22 abyss^ ndevos: oo, thank you;)
11:23 ndevos :)
11:27 ctria joined #gluster
11:28 tokik joined #gluster
11:39 edward1 joined #gluster
11:39 Slash joined #gluster
11:43 Philambdo joined #gluster
11:45 spandit joined #gluster
11:45 ndarshan joined #gluster
11:47 aravindavk joined #gluster
11:50 Philambdo joined #gluster
11:51 kanagaraj joined #gluster
11:51 kshlm joined #gluster
11:52 RameshN joined #gluster
11:52 kkeithley1 joined #gluster
11:54 ppai joined #gluster
11:59 franc joined #gluster
11:59 franc joined #gluster
12:03 sahina joined #gluster
12:05 shyam joined #gluster
12:06 rgustafs joined #gluster
12:19 glusterbot New news from newglusterbugs: [Bug 1063230] DHT - rebalance - when any brick/sub-vol is down and rebalance is not performing any action(fixing lay-out or migrating data) it should not say 'Starting rebalance on volume has been successful' . <https://bugzilla.redhat.com/show_bug.cgi?id=1063230>
12:24 Philambdo joined #gluster
12:25 khushildep joined #gluster
12:29 prasanth|mtng joined #gluster
12:32 NuxRo ndevos: great news
12:32 NuxRo thanks for all your work
12:33 NuxRo ndevos: btw check the "Who committed this?" email on the ACS ml
12:33 NuxRo concerning your commit
12:35 rjoseph joined #gluster
12:37 ndevos NuxRo: oh, hmm, strange, it surely is in the patches... maybe the review got it stripped somehow
12:38 ndarshan joined #gluster
12:39 ppai joined #gluster
12:39 prasanth joined #gluster
12:39 spandit joined #gluster
12:50 hagarth joined #gluster
12:57 NuxRo ndevos: looks like it was Wido :)
12:59 davinder joined #gluster
13:06 diegows joined #gluster
13:18 rastar joined #gluster
13:23 benjamin_____ joined #gluster
13:29 rjoseph joined #gluster
13:33 eclectic joined #gluster
13:47 rwheeler joined #gluster
13:51 ndarshan joined #gluster
13:53 B21956 joined #gluster
14:03 bennyturns joined #gluster
14:03 bennyturns |afk
14:11 japuzzo joined #gluster
14:13 cfeller joined #gluster
14:14 sroy joined #gluster
14:20 chirino joined #gluster
14:23 kmai007 joined #gluster
14:24 str8uplinux joined #gluster
14:25 str8uplinux trying to download the gluster-deploy tar file, but I don't see any place to download it here: https://forge.gluster.org/gluster-deploy
14:25 glusterbot Title: gluster-deploy - Gluster Community Forge (at forge.gluster.org)
14:30 theron joined #gluster
14:30 rgustafs joined #gluster
14:32 rgustafs joined #gluster
14:33 dbruhn joined #gluster
14:35 xavih str8uplinux: go to "Source tree" and there will be a link to download master branch as a tar.gz file
14:40 hybrid512 joined #gluster
14:40 dbruhn johnmark_, There was a ton of Gluster talk at SCALE. Thought you would like to hear that.
14:41 str8uplinux xavih: i didn't that, but the only thing in the tar.gz file is a singe file called master that has no file extension.
14:42 str8uplinux am i missing something?
14:42 xavih str8uplinux: rename the file to master.tar.gz
14:42 xavih str8uplinux: for some reason it loses the file extension while downloading
14:43 str8uplinux so there is a tar.gz within the tar.gz?
14:43 xavih no, simply rename the downloaded file
14:43 jmarley joined #gluster
14:43 jmarley joined #gluster
14:43 chirino joined #gluster
14:43 str8uplinux xavih, thanks!
14:44 xavih str8uplinux: yw
14:50 benjamin_____ joined #gluster
14:51 lmickh joined #gluster
14:52 merrittk joined #gluster
14:56 merrittk Would some be able to assist me with a splitbrain issue, not sure how to resolve. I am using glusterfs 3.4.2.
14:56 merrittk I get the following in the logs [2014-02-25 14:54:04.723897] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-shared_2-replicate-0: Unable to self-heal contents of '<gfid:00000000-0000-0000-0000-000000000001>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 4 ] [ 2 0 ] ]
14:56 vpshastry joined #gluster
14:56 merrittk and the output from gluster volume heal shared_2 info
14:57 merrittk Gathering Heal info on volume shared_2 has been successful
14:57 merrittk Brick host1:/brick_2/stash-data
14:57 merrittk Number of entries: 1
14:57 merrittk Brick host2:/brick_2/stash-data
14:57 merrittk Number of entries: 1
14:57 merrittk after number of entries is /
14:58 str8uplinux left #gluster
14:59 rpowell joined #gluster
15:00 dbruhn merrittk, check your ./glusterfs directory in there is a link to 00000000-0000-0000-0000-000000000001
15:01 dbruhn sometimes the link gets broken
15:01 dbruhn .glusterfs is at the root of each brick
15:02 merrittk yeah ok, I found it, but dont see a link for 00000000-0000-0000-0000-000000000001
15:03 merrittk I have this: 00/00/00000000-0000-0000-0000-000000000001
15:03 dbruhn should look something like this with a "ls -la"
15:03 dbruhn lrwxrwxrwx   1 root root       8 Jan 22 13:30 00000000-0000-0000-0000-000000000001 -> ../../..
15:04 dbruhn check each brick and make sure it still does
15:05 merrittk I dont have it on any of the bricks, just a bunch of hex dirs and indices and landfill.
15:06 zerick joined #gluster
15:10 jmarley joined #gluster
15:12 kmai007 merrittk:do a 'find <brick_path> -name 00000000-0000-0000-0000-000000000001'
15:12 kmai007 the results you should be able to do an ls -l <path>
15:13 bugs_ joined #gluster
15:15 zerick joined #gluster
15:15 merrittk ls -l /brick_2/stash-data/.glusterfs/00/00/00000000-0000-0000-0000-000000000001
15:15 merrittk lrwxrwxrwx 1 root root 8 Feb 24 17:18 /brick_2/stash-data/.glusterfs/00/00/00000000-0000-0000-0000-000000000001 -> ../../..
15:15 merrittk its the same on both bricks
15:17 dbruhn sorry I misread what you had in chat and though it was saying / was in split brain
15:18 dbruhn hmm actually it is saying it is out of waco
15:18 dbruhn s/waco/wack/
15:18 glusterbot What dbruhn meant to say was: hmm actually it is saying it is out of wack
15:18 merrittk it does say /
15:18 merrittk #Gathering Heal info on volume shared_2 has been successful
15:18 merrittk #Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
15:18 merrittk #Number of entries: 1
15:18 merrittk #/
15:18 merrittk #
15:18 merrittk #Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
15:18 merrittk #Number of entries: 1
15:18 merrittk #/
15:19 dbruhn is stash-data your volume?
15:19 dbruhn run this on each of your bricks "getfattr -m . -d -e hex /brick_2/stash-data"
15:21 dbruhn what's the output
15:21 merrittk # file: brick_2/stash-data
15:21 merrittk trusted.afr.shared_2-client-0=0x000000000000000000000000
15:21 merrittk trusted.afr.shared_2-client-1=0x000000000000000400000000
15:21 merrittk trusted.gfid=0x00000000000000000000000000000001
15:21 merrittk trusted.glusterfs.dht=0x000000010000000000000000ffffffff
15:21 merrittk trusted.glusterfs.volume-id=0x1a3305c44a724d298396742e1edbc04e
15:21 merrittk # file: brick_2/stash-data
15:21 merrittk trusted.afr.shared_2-client-0=0x000000000000000200000000
15:21 merrittk trusted.afr.shared_2-client-1=0x000000000000000000000000
15:21 merrittk trusted.gfid=0x00000000000000000000000000000001
15:21 merrittk trusted.glusterfs.dht=0x000000010000000000000000ffffffff
15:21 merrittk trusted.glusterfs.volume-id=0x1a3305c44a724d298396742e1edbc04e
15:25 jobewan joined #gluster
15:27 ikk joined #gluster
15:28 dbruhn Do those directories contain all of the same directory information?
15:28 dbruhn I am only concerned about the directories directly under it
15:28 dbruhn they should be the same
15:29 cp0k hey guys, Im having trouble probing a new peer, when I probe it gluster says success, but on the newly probed peer I see: Number of peers: 1 and State: Peer Rejected (Connected)
15:29 merrittk looks the same to me
15:29 dbruhn merrittk, permissions the same?
15:30 dbruhn cp0k, iptables or selinux?
15:30 cp0k instead of seeing the full peer list like on the other nodes
15:30 cp0k iptables
15:30 dbruhn is iptables running?
15:30 cp0k yes
15:30 dbruhn can you try and disable it and see if the issue goes away?
15:30 merrittk perms are the same
15:31 cp0k sure, sec
15:31 cp0k dbruhn: only port I am firewalling is 22 though
15:31 jbrooks joined #gluster
15:31 RameshN joined #gluster
15:32 dbruhn merrittk, "setfattr -n trusted.afr.shared_2-client-0 -v 0x000000000000000000000000 brick_2/stash-data" for each of those trusted.afr.shared attributes might clear it up
15:32 cp0k dbruhn: on the new peer I probed: [glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: ade0d801-618e-4403-9199-7aa126f8a9b5, host: 10.0.144.214, port: 0
15:32 cp0k port 0?
15:33 dbruhn cp0k, are you seeing anything in the logs on either side that would give you an indication of failure
15:34 merrittk dbruhn: should i run it for both attributes on both nodes?
15:34 cp0k dbruhn: one sec, let me check
15:34 wushudoin joined #gluster
15:35 bala joined #gluster
15:36 cp0k [2014-02-25 15:36:14.357862] I [glusterd-handler.c:2280:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.0.144.214 (24007)
15:37 cp0k [2014-02-25 15:36:14.524995] E [glusterd-utils.c:4255:glusterd_brick_start] 0-management: Could not find peer on which brick 10.0.144.211:/gluster/4 resides
15:37 dbruhn merrittk, yeah, those attributes only show up with something has been in split-brain
15:37 kmai007 cp0k: for kicks restart your glusterd service on the new brick
15:37 dbruhn a self heal might have resolved the issue already, I am really not sure, but i've seen weird things with split brain stuff on directories
15:37 merrittk I ran it on both hosts, both hosts show:
15:37 merrittk # file: brick_2/stash-data/
15:37 merrittk trusted.afr.shared_2-client-0=0x000000000000000000000000
15:37 merrittk trusted.afr.shared_2-client-1=0x000000000000000000000000
15:37 merrittk trusted.gfid=0x00000000000000000000000000000001
15:37 merrittk trusted.glusterfs.dht=0x000000010000000000000000ffffffff
15:37 merrittk trusted.glusterfs.volume-id=0x1a3305c44a724d298396742e1edbc04e
15:38 merrittk but the issue still seems to be there.
15:38 dbruhn hmm ok
15:38 dbruhn there are no hidden directories on the root of your bricks?
15:39 merrittk nop
15:39 merrittk e
15:39 dbruhn and /brick_2/stash-data is the directory that is the root of the brick?
15:40 kmai007 merrittk: does any of your clients report split brain of the volume?
15:40 merrittk dbruhn: yes
15:42 merrittk my client shows: [2014-02-25 14:27:17.069722] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-shared_2-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 4 ] [ 2 0 ] ]
15:42 cp0k kmai007: restarted glusterd on the new brick and now seeing stuff like this on that new brick:
15:42 cp0k [2014-02-21 21:34:08.390206] E [client-handshake.c:1742:client_query_portmap_cbk] 0-th-tube-storage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
15:43 cp0k gluster volume info only shows status for the localhost (new peer I am probing)
15:44 cp0k only way I was able to get this to work in the past was to manipulate /var/lib/glusterd/peers, but that seemed like a dirty hack which, while everything worked out, the rest of the peers in the pool didnt see the host I added in the odd fashion
15:44 kmai007 cp0k: i've experienced that on a new peer.
15:44 kmai007 i blieve there is a feature that will migrate the vol file to the new brick
15:44 kmai007 volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
15:45 kmai007 i haven't done it enough to recall my steps
15:45 cp0k kmai007: great! happy to hear that you have experience with it :)
15:46 cp0k kmai007: so 'gluster volume sync 10.0.144.115 th-tube-storage all' in my case?
15:46 cp0k kmai007: do I execute this on the new peer that has already been probed?
15:46 kmai007 i don't recall, if its saying force all peers to pull the vol file from 10.0.144.115
15:47 gmcwhistler joined #gluster
15:47 kmai007 i think if you search the web you'll see examples of how it was successfully used
15:47 wica Hi, if I kill 1 glusterfsd proces for 1 brick, what will happen?
15:48 failshell joined #gluster
15:48 cp0k kmai007: sounds good, thanks for pointing me in the right direction at least
15:48 wica or it it possible to mark a brick has bad ?
15:48 kmai007 cp0k: http://gluster.org/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server
15:48 glusterbot Title: Gluster 3.2: Brick Restoration - Replace Crashed Server - GlusterDocumentation (at gluster.org)
15:50 zerick joined #gluster
15:55 merrittk dbruhn / kmai007: Any ideas on what I can try next? Anyway i can tell gluster that node1 is good and just replicate everything to node2 again?
15:55 kmai007 how often does your client log the possible split brain?
15:56 kmai007 from your getfattr output it looks to have the correct attributes on both bricks
15:56 kmai007 pick another file inside your volume and see if it has the same attribute hex from getfattr
15:56 daMaestro joined #gluster
15:58 wica not possible to mark a brick as not to use?
15:58 merrittk it has been a while since the client has logged a splitbrain.
15:59 kmai007 if everything appears functional and your files are not showing file not found in the clients; 'gluster volume heal <vol> info split-brain' output can only be reset if you restart glusterd
15:59 kmai007 that is what i've learned from JoeJulian
16:00 kmai007 but before then
16:00 kmai007 what does 'gluster volume heal <vol> info' show you?
16:00 kmai007 that is a live running list
16:01 merrittk Everything looks fine from getfattr, i checked a few files/dirs and everything matches
16:01 merrittk #Gathering Heal info on volume shared_2 has been successful
16:01 merrittk #Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
16:01 merrittk #Number of entries: 1
16:01 merrittk #/
16:01 merrittk #
16:01 merrittk #Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
16:01 merrittk #Number of entries: 1
16:01 merrittk #/
16:02 kmai007 does 'gluster volume heal <vol> info split-brain' report anything?
16:02 merrittk yes, its long, should i paste it?
16:02 kmai007 nope
16:02 zerick joined #gluster
16:02 kmai007 that should be your focus tho'
16:03 kmai007 i spent the past 5 days working down 1023 max number of split brains
16:03 kmai007 files
16:03 merrittk Number of entries: 14
16:03 merrittk , vs Number of entries: 11
16:03 kmai007 attack them on a per brick basis
16:03 kmai007 http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/
16:03 glusterbot Title: Fixing split-brain with GlusterFS 3.3 (at joejulian.name)
16:03 merrittk they are all 2014-02-25 15:34:05 /
16:03 merrittk just with diff timestamps.
16:03 kmai007 yuck
16:04 kmai007 i don't know how to act on /
16:04 kmai007 is this a replica ?
16:04 merrittk yes
16:04 merrittk and i am certain the data is intact on both replicas.
16:04 cp0k did JoeJulian release his new notes on split brain stuff yet? last I spoke to him he said he will be releasing it yesterday (Monday)
16:05 kmai007 here is a useful doc. from redhat https://access.redhat.com/site/sites/default/files/attachments/rhstorage_split-brain_20131120_0.pdf
16:05 kmai007 cp0k: no i haven't seen or heard any updates for JoeJulian
16:05 kmai007 merrittk: attempt to fix the split-brain on the least amount of files reported by a brick
16:06 kmai007 you may find that they are repeats with different time stamps
16:06 merrittk I have read that redhat doc, prior to trying irc
16:06 cp0k to fix my split brain , I had just nuked all the gfid's in the list
16:06 cp0k that fixed me up :)
16:07 kmai007 <----same
16:07 cp0k now I run gluster volume heal volname info split-brain over and over with a smile on my face
16:07 cp0k love seeing number of entries: 0
16:08 cp0k however I am still seeing entries (89 at the max) when doing gluster volume heal volname info
16:08 cp0k should I take care of those as well prior to adding my new bricks?
16:09 cp0k as I understand it, the 'info' output is a list of files that need healing....but it seems that number is not going down on its own, therefor gluster will never fix them?
16:09 merrittk kmai007: I restarted glusterd as you said earlier and it seems fine now :/
16:10 merrittk Thank you so much for the help kmai007 / dbruhn
16:11 kmai007 merrittk: np,  glad to see another happy sys admin
16:11 kmai007 cp0k: that is a tough one, i've never had to do that operation....
16:11 merrittk bleh, i spoke to soon, gluster volume heal shared_2 info split-brain shows  its fine, gluster volume heal shared_2 info is the same
16:12 kmai007 did u restart it on that brick?
16:12 kmai007 glusterd restart per brick, to reset vol heal info split-brain output
16:12 merrittk i brought gluster down all both hosts, then started it again
16:13 merrittk gluster volume heal shared_2 info
16:13 merrittk Gathering Heal info on volume shared_2 has been successful
16:13 merrittk Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
16:13 merrittk Number of entries: 1
16:13 merrittk Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
16:13 merrittk Number of entries: 1
16:13 merrittk Gathering Heal info on volume shared_2 has been successful
16:13 merrittk Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
16:13 merrittk Number of entries: 0
16:13 merrittk Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
16:13 merrittk Number of entries: 0
16:14 kmai007 info means just that, until it gets moved to the other buckets of heal-failed or split-brain
16:15 merrittk *confused*
16:17 kmai007 in my experience i've only focused on split-brain info
16:19 merrittk So i can safely ignore the output of gluster volume heal shared_2 info, assuming split-brain reports 0?
16:20 dbruhn merrittk, sorry I had to step away for a bit
16:20 dbruhn looks like it's resolved?
16:21 cp0k kmai007: so Im checking all my peers now to make sure the data in /var/lib/glusterd is all consistent....Im a little nervous about running the vol sync command on a production env without ever using it in the past
16:22 cp0k kmai007: liek I said, by manipulating /var/lib/glusterd/peers in the past, I was able to get the new peer added up. Only trouble is that the rest of the peers dont see it in the loop
16:22 cp0k as the addition was not made via gluster peer probe
16:22 merrittk np, to be honest i am not sure. If i can safely ignore ignore:
16:22 merrittk $ gluster volume heal shared_2 info
16:22 merrittk #Gathering Heal info on volume shared_2 has been successful
16:22 merrittk #Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
16:22 merrittk #Number of entries: 1
16:22 merrittk #/
16:22 merrittk #Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
16:22 merrittk #Number of entries: 1
16:23 merrittk #/
16:23 rastar joined #gluster
16:23 merrittk because this reports no errors:
16:23 merrittk $ gluster volume heal shared_2 info split-brain
16:23 merrittk #Gathering Heal info on volume shared_2 has been successful
16:23 merrittk #Brick htf-aihceprodb01.fnb.co.za:/brick_2/stash-data
16:23 merrittk #Number of entries: 0
16:23 merrittk #
16:23 merrittk #Brick htf-aihceprodb02.fnb.co.za:/brick_2/stash-data
16:23 merrittk #Number of entries: 0
16:23 merrittk then its resolved :)
16:23 jag3773 joined #gluster
16:24 dbruhn Well, it's not in split-brain at the moment, so that's good
16:25 dbruhn typically to trigger a heal you would stat the file or directory in the volume.... now sure how you would do that at the root.
16:25 mrverrall joined #gluster
16:25 cp0k dbruhn / kmai007: is it bad to have "gluster volume heal volname info" entries prior to adding new bricks?
16:25 dbruhn So.. if it's not causing you any issues, and you aren't seeing a mess in the logs, it's seems to be a moot report.
16:26 kmai007 cp0k: i hear ya, its nerve wrecking, if you can, wait for the experts for their input
16:26 dbruhn cp0k, that wouldn't cause the issue you are seeing now, but always suggest correcting any issues with your volume before trying to expand
16:26 merrittk ok, ill ignore it for now then and see how it goes. Thank you.
16:26 mrverrall Hi I'm looking tomount my gluster volume in two different locations. My question is does gluster handle the file locking nativly or do I need to use an fs like ocfs2 when creating the bricks?
16:27 cp0k I agree, might as well clean it up fully before adding the bricks
16:27 cp0k as for the issue with adding the new bricks, Im sure its just something silly in /var/lib/glusterd on one of my 30 existing peers thats knocking things off center
16:27 cp0k (This was the case with me getting exit code 146 when running a split-brain report)
16:28 dbruhn as kmai007 was saying earlier there is a command to copy the config from a known good server, but I am not sure what it is right off.
16:28 cp0k a number of my peers had itself probed
16:28 cp0k dbruhn: seems to be 'gluster volume sync'
16:29 kmai007 cp0k: can you spin up a dev gluster
16:29 cp0k kmai007: I already have one up
16:29 dbruhn mrverrall,
16:29 dbruhn Do clients communicate with each other?
16:29 dbruhn No clients do not communicate with each other.
16:29 dbruhn Do servers communicate with each other?
16:29 dbruhn Yes, glusterd daemons communicate directly for administration purposes.
16:29 dbruhn Additionally, each server re-shares the volume via nfs through a client process. That client process connects to it's volume servers.
16:29 dbruhn Finally, a rebalance also uses the client to connect to a volume's servers to perform the rebalance.
16:29 dbruhn What happens if a GlusterFS brick crashes?
16:29 dbruhn You treat it like any other storage server. The underlying filesystem will run fsck and recover from crash. With journaled file system such as Ext3 or XFS, recovery is much faster and safer. When the brick comes back, glusterfs fixes all the changes on it by its self-heal feature.
16:29 kmai007 and see if you can try to run a volume sync  and see what it does?
16:29 dbruhn What about deletion self/auto healing?
16:29 dbruhn With auto healing only file creation is healed. If a brick is missing because of a disk crash re-creation of files is ok but if it's a temporary network problem synchronizing deletion is mandatory.
16:29 dbruhn See also Gluster 3.2: Triggering Self-Heal on Replicate.
16:29 dbruhn Can I add or remove a storage node while GlusterFS is online?
16:29 dbruhn Dynamic volume management tools to add/remove bricks online were released in v3.1.
16:29 dbruhn Can I directly access the data on the underlying storage volumes?
16:30 dbruhn If you are just doing just read()/access()/stat() like operations, you should be fine. If you are not using any new features (like quota/geo-replication etc etc) then technically, you can modify (but surely not rename(2) and link(2)) the data inside.
16:30 dbruhn Note that this is not tested as part of gluster's release cycle and not recommended for production use.
16:30 dbruhn What happens in case of hardware or GlusterFS crash?
16:30 theron_ joined #gluster
16:30 dbruhn You don't risk any corruption. How ever if the crash happened in the middle of your application writing data, the data in transit may be lost. All file systems are vulnerable to such loses.
16:30 dbruhn Metadata Storage - When using striping (unify), how does the file data get split?
16:30 dbruhn Individual files are never split and stored on multiple bricks, rather, the scheduling algorithm you specify is used to determine which brick a file is stored on.
16:30 dbruhn Metadata Storage - When using striping (unify), how/where is the metadata kept?
16:30 dbruhn As said earlier, there is no metadata in unify (in whole GlusterFS itself). Unify keeps its namespace cache in an separate namespace volume.
16:30 dbruhn How to make GlusterFS secure?
16:30 dbruhn GlusterFS as of now supports only IP/port based authentication. You specify a range of IP addresses separately for clients and management nodes to allow access. Client side port is always restricted to less than 1024 to ensure only root can perform management operations including mount/umount. New GNU TLS (secure certificate) based authentication is under development. We are also planning to implement encryption translator in the upcoming release. Till the
16:30 dbruhn n you can even stunnel GlusterFS connections.
16:30 dbruhn Here is one article about setting up Encrypted Network between client and server.
16:30 dbruhn How do I mount/umount GlusterFS?
16:30 dbruhn Refer to Mounting a GlusterFS Volume.
16:30 dbruhn Do I need to synchronize UIDs and GIDs on all servers using GlusterFS ?
16:30 dbruhn No. Only clients machines need to be synchronized, since the access control is done on the client's side VFS layer.
16:30 kmai007 are you pasting something?
16:30 dbruhn Do I need to synchronize time on all servers using GlusterFS ?
16:30 dbruhn Yes. You can use NTP (Network Time Protocol) client to do this if your hosting environment does not do this for you (for example Amazon EC2 already does this). Keeping all server time in sync is a good thing. Few translators like io-cache which works based on mtime may not work properly otherwise.
16:30 gmcwhistler joined #gluster
16:30 dbruhn Simple example of NTP command:
16:30 dbruhn bash# /usr/sbin/ntpdate pool.ntp.org
16:30 dbruhn How do I add a new node to an already running cluster of GlusterFS
16:30 dbruhn Yes, you can add more bricks in your volume specification file and restart GlusterFS (re-mount). Its schedulers (alu) are designed to balance the file system data as you grow.
16:30 dbruhn For releases after 1.3.0-pre5
16:31 dbruhn Just add the extra node in unify's or DHT's subvolumes list, and restart the GlusterFS, the directory structure is automatically replicated in the new server :D The much desired self-heal property of unify solves the burden of manually maintaining equal directory structure in all the servers before mount.
16:31 dbruhn Note: We are planning to add on-the-fly addition of storage bricks in our next release. The above steps will be taken care automatically.
16:31 dbruhn How do I add a new AFR namespace brick to an already running cluster?
16:31 dbruhn The question is not quite clear. Right now, GlusterFS doesn't support on-the-fly change of volfiles to add volumes. You need to edit volfile, add afr volume, stop glusterfs process, start it with new volfile to achieve this.
16:31 dbruhn Loop mounting image files stored in GlusterFS file system
16:31 dbruhn To mount one image file stored in glusterfs file system, you have to disable the direct-io in the glusterfs mount. to do this with GlusterFS use the following command:
16:31 dbruhn #glusterfs -f <your_spec_file>  --disable-direct-io-mode /<mount_path>
16:31 dbruhn After that you can use your glusterfs file system mounted on /<mount_path> to store your images. If you disable direct-io you can use glusterfs to store xen virtual machines virtual block device as files. Xen + Live Migration works fine using the option above.
16:31 dbruhn How do you allow more than one IP in auth.addr?
16:31 dbruhn Q: If you can only have one auth.addr line in a config, how do you allow 127.0.0.1 as well as a 192.168.* range?
16:31 dbruhn A: Make your auth.addr.<volumename>.allow look like this:
16:31 dbruhn option auth.addr.<volumename>.allow 127.0.0.1,192.168*
16:31 dbruhn Note the comma separated ip address patterns.
16:31 dbruhn Stripe behavior not working as expected
16:31 dbruhn Q: Striping doesn't work well. I made a file of 4MB with 'option block-size 2MB', but on my two servers the file is distributed like this:
16:31 dbruhn PC1: file = 2MB
16:31 dbruhn PC2: file = 4MB
16:31 dbruhn A: GlusterFS's stripe translator saves files as files in backend, but with a filesystem holes. 'ls' doesn't understand the filesystem holes, but du does. Please check the disk used by the file with the help of du command. That should show 2MB each.
16:31 dbruhn Duplicate volume name in volfile
16:31 dbruhn Q: Is it possible to use the same brick name several times in the same glusterfs-server.vol like in the example below?
16:31 dbruhn volume brick
16:32 dbruhn type storage/posix
16:32 dbruhn option directory /dfslarge
16:32 dbruhn end-volume
16:32 dbruhn volume brick
16:32 dbruhn type storage/posix
16:32 dbruhn option directory /dfssmall
16:32 dbruhn end-volume
16:32 dbruhn A: No, volume name should be unique across the volume file. GlusterFS process will be erroring out in this case.
16:32 dbruhn File location
16:32 dbruhn Q: For example, I have 3 servers(no ssh) in Unify mode with RR scheduler and I've uploaded some file. How can I find out on which brick a file is located?
16:32 dbruhn A: Currently there is no mechanism to find out the files at server. This should be made available in next releases.
16:32 dbruhn What is GlusterFS scheduler?
16:32 dbruhn The GlusterFS scheduler handles load-balancing and high-availability in clustered mode when unify translator is used. You select a scheduler of your choice in your "unify" volume. Check this link for more information about type of schedulers, their options, benefits of using them etc..
16:32 dbruhn Servers with Multiple Purposes
16:32 dbruhn Q: Can a cluster of servers be used for multiple purposes, e.g. run GlusterFS + Apache/PHP?
16:32 dbruhn A: Dedicated servers are recommended due to security and performance concerns. How ever there are no restrictions for GlusterFS to coexist with other services (such as Apache or MySQL).
16:32 dbruhn How can I improve the performance of reading many small files?
16:32 dbruhn Use the NFS client. For reading many small files, i.e. PHP web serving, the NFS client will perform much better.
16:32 dbruhn That that for a write-heavy load the native client will perform better.
16:32 dbruhn gahh, sorry
16:32 dbruhn mrverrall, File level locking is handled distributedly across the bricks using features/posix-locks translator. GlusterFS supports both fcntl() and flock() calls.
16:32 dbruhn sorry for the flood, grabbed too much info out of the wiki on accident
16:32 dbruhn I did
16:33 gmcwhist_ joined #gluster
16:34 hybrid512 joined #gluster
16:34 mrverrall hehe, no problem, I though I was getting an automated FAQ for asking dumb question :
16:35 mrverrall So in shourt I can mount an export in as many locations as I like just using XFS bricks and glusterfs mounts?
16:35 rpowell left #gluster
16:38 dbruhn yep
16:38 mrverrall You have no idea how happy that makes me :)
16:44 vpshastry joined #gluster
16:45 kmai007 any idea why this is logged on my client: [2014-02-25 16:25:20.167257] W [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (No data available), peer (167.132.88.124:24007)
16:45 kmai007 when i check brick 167.132.88.124
16:45 vpshastry left #gluster
16:45 kmai007 i see a connection still established
16:45 ColPanik I'm in the middle of restarting gluster on a node.  The cli on this node reports only seeing one of two volumes in "gluster volume info".  If I query the 'missing' volume directly by name, it tells me that the volume doesn't exist.  However, I can see that there's a glusterfsd process with this volume as it's --volfile argument that's been heavily using CPU for about the last 30 minutes
16:46 ColPanik there are a lot of files on the volume, so it could just be taking a long time to reconcile files vs metadata or something
16:47 ColPanik but wondering if people could add any experience there
16:47 ColPanik I've seen the cpu % drop steadily
16:47 aquagreen joined #gluster
16:48 kmai007 how do you query the volume directly?
16:48 ColPanik also, I can see in strace that the process is doing a ton of reads
16:49 ColPanik kmai007: gluster volume status VOLNAME
16:49 ColPanik kmai007: (also, hi again :)  )
16:50 kmai007 ColPanik: sup, so in 'gluster volume status <vol>' do you know if that port is active?
16:50 kmai007 it should spit out the port its listening on
16:51 kmai007 basically 24007 is the management port gluster uses to tell the volumes what do
16:52 kmai007 could it be heavily used hence why the CPU is busy, access from clients
16:53 ColPanik if I run 'volume status VOLNAME' it tells me that my volume doesn't exist yet, so no port information.  This might seem like the volume isn't set up or configured correctly, but a) it was there before a reboot b) there's a process going cpu-crazy that's pointed at the volume in question
16:53 kmai007 wow today must be a volfile sync day
16:53 kmai007 last week was split-brain week
16:53 ColPanik haha, funny how it all moves in lock step
16:54 kmai007 does it have any volume information about any volumes?
16:54 kmai007 gluster volume status
16:54 ColPanik yea, there's a smaller volume that's showing up correclt
16:54 kmai007 should spit out everything
16:54 kmai007 so every brick reports the same thing?
16:55 kmai007 or just a single brick?
16:55 ColPanik no, the other brick sees the volume in question
16:59 kmai007 web search volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
16:59 kmai007 that brick is probably having a jacked up volfile
16:59 kmai007 i've only used it once
16:59 kmai007 that vol file should get yoru brick back to normalacy
16:59 kmai007 with its peers
17:00 kmai007 other than that, i'm not sure how else to proceed
17:03 KyleG joined #gluster
17:03 KyleG joined #gluster
17:08 sprachgenerator joined #gluster
17:09 rpowell1 joined #gluster
17:11 mrverrall one more quickie, is the preference nfs or the native clisterfs (fuse?)
17:11 dbruhn depends on what you are doing
17:11 mrverrall Yeah I realise that was a touch open ended
17:12 dbruhn the native client has some resiliency that NFS doesn't, NFS is of course ez mode, NFS has a single point of failure by default
17:12 mrverrall in short, providing a storage backend to a web service that has several frontends
17:12 dbruhn NFS is better for small file operations, so if you are trying to squeak some performance out of a system it might give you that few percent improvement
17:12 gmcwhistler joined #gluster
17:13 mrverrall uuber tweaking is not the order of the day :)
17:13 dbruhn well then the fuse client is probably the way to go, as it will handle a dropped server better
17:13 mrverrall many thanks, you have been most helpful :)
17:14 semiosis mrverrall: just a wild guess, but this ,,(php) info might be useful
17:14 glusterbot mrverrall: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
17:14 glusterbot --fopen-keep-cache
17:14 dbruhn np, obviously test test test test test on all of this
17:15 hybrid512 joined #gluster
17:15 mrverrall The good news is that code is all stored on the front ends, only files are served from gluster (well that is the plan)
17:15 kmai007 semiosis: where can i find more info about those mount attributes --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
17:15 mrverrall and mostly apc cached.
17:16 cjanbanan joined #gluster
17:16 mrverrall considering sticking some varnish in front of it all to boot
17:17 semiosis mrverrall: using autoloading (hopefully your framework does already!) will give a huge performance boost vs lots of require calls
17:17 mrverrall one thing at a time though and obviously test test test :)
17:17 semiosis varnish is good!
17:17 semiosis kmai007: never heard of them, sorry
17:17 semiosis oh, well i've heard of attribute-timeout
17:17 semiosis but idk where to find docs about them
17:17 kmai007 ok just saw what glusterbot spit out as #2 option
17:18 semiosis @mount options
17:18 semiosis eh, was worth a try
17:18 vpshastry joined #gluster
17:18 mrverrall semiosis: looks like autoloading is used :)
17:19 semiosis then you should be fine.  in the bad old days before autoloading was commonplace most php apps would require every file in the framework for each load, even files that weren't used
17:19 semiosis this caused poor performance on glusterfs
17:19 semiosis but with autoloading & apc it works pretty well
17:20 mrverrall hopefully moot as my code is local anyway :)
17:20 hagarth joined #gluster
17:20 semiosis ah, code local, data on gluster? thats a good plan!
17:20 mrverrall aye :)
17:22 mrverrall So do people run vrrp so the clients always have a live ip to connect to?
17:23 mrverrall I know we can loose a server once it's connected, but what about before the mount
17:23 semiosis see ,,(mount server)
17:23 glusterbot The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrdns
17:23 semiosis see also ,,(rrdns)
17:23 glusterbot You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
17:24 mrverrall oh round robin works fine then
17:24 mrverrall might be more reliable then keepalived :\
17:24 mrverrall Ok, now I am asking dumb questions. Thanks again.
17:30 diegows joined #gluster
17:30 redbeard joined #gluster
17:34 larsks Hello all, a repeat from yesterday: I'm getting erroneous "permission denied" errors from files on a gluster (fuse) mount that go away when the process that has the file open exits.  Has anyone seen this before?
17:35 cjanbanan joined #gluster
17:35 larsks Acess to the same file on another client works just fine.
17:36 kmai007 are you describing concurrent access to the same file by 2 different clients
17:36 larsks kmai007: Nope.  There is a file (disk image for a vm) that returns "permission denied" errors on any attempt to access it *on the same client as the vm*.
17:37 larsks If I kill the qemu instance, it starts to work -- and *continues* to work if I restart the vm.
17:37 larsks (whereby "work" I mean "does not return errors on read attempts")
17:37 Matthaeus joined #gluster
17:39 jag3773 joined #gluster
17:40 asku left #gluster
17:40 Mo_ joined #gluster
17:42 madphoenix joined #gluster
17:42 vpshastry joined #gluster
17:43 madphoenix hi all, hoping somebody can help me with a mistake.  i accidentally ran replace-brick without "start", and it instead did commit-force, which seems to have just offlined the brick instead of starting a migration.  is there an easy way to add it back, or am i looking at a restore situation here?
17:44 KyleG left #gluster
17:46 madphoenix s/replace-brick/remove-brick/
17:46 glusterbot What madphoenix meant to say was: hi all, hoping somebody can help me with a mistake.  i accidentally ran remove-brick without "start", and it instead did commit-force, which seems to have just offlined the brick instead of starting a migration.  is there an easy way to add it back, or am i looking at a restore situation here?
17:47 kmai007 larsks: oh yeh, you're running kvm, from yesterday
17:47 cjanbanan joined #gluster
17:47 kaptk2 joined #gluster
17:47 larsks kmai007: Yup, that's me.
17:59 failshel_ joined #gluster
18:02 cjanbanan joined #gluster
18:04 jiqiren joined #gluster
18:06 vpshastry left #gluster
18:07 jbrooks joined #gluster
18:10 cfeller_ joined #gluster
18:10 divbell_ joined #gluster
18:10 jurrien__ joined #gluster
18:11 sputnik13net joined #gluster
18:17 maksim joined #gluster
18:19 bennyturns joined #gluster
18:20 lmickh joined #gluster
18:21 maksim Hi guys, I run a site that helps devs called airpair.com.  We have a member who's looking to talk to someone about performance tuning GlusterFS and is ready to pay $260/hr for 1-3 hours of help via video chat and screen sharing.  Would anyone here be interested in something like this?  Check out http://www.airpair.com/review/530cdc52d2b467020000002d for more info.
18:21 glusterbot Title: AirPair | storage request (at www.airpair.com)
18:22 masterzen_ joined #gluster
18:23 fidevo joined #gluster
18:23 fidevo joined #gluster
18:30 kmai007 performance tuning is difficult without truely understanding their purpose
18:31 primusinterpares joined #gluster
18:32 maksim kmai - they are looking to connect with someone who has deep understanding of GlusterFS
18:33 JoeJulian ~php | maksim
18:33 glusterbot maksim: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizing-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH --fopen-keep-cache
18:34 kmai007 JoeJulian:hi, i wanted to ask you about #2, where can i find more info regarding how to use it?
18:35 maksim Joe - thanks.  I'm trying to find someone who has deep knowledge of Gluster who'd have 1-3 hrs to spend helping out with this performance issue.  They'd get paid $260/hour
18:37 dewey joined #gluster
18:37 vpshastry1 joined #gluster
18:38 cjanbanan joined #gluster
18:40 zaitcev joined #gluster
18:40 dbruhn joined #gluster
18:43 fidevo joined #gluster
18:43 askb joined #gluster
18:48 JoeJulian maksim: Yeah, got that. I'm more of a facilitator. Plus, if you optimize your web stack, the file system shouldn't matter much.
18:48 JoeJulian kmai007: I got that from one of the devs and haven't had the time to browse the source to find out any more.
18:49 cp0k JoeJulian: hey, have you had a chance to publish your new fancy paper on split-brain ? :)
18:49 JoeJulian Sorry, no. Had to drive across the state and back yesterday.
18:50 zwevans joined #gluster
18:50 cp0k JoeJulian: no worries, just curious...my split brain issue is resolved anyway for now :)
18:50 JoeJulian yay!
18:50 glusterbot New news from newglusterbugs: [Bug 1069840] GlusterFS rpm compilation fails on CentOS 5.x <https://bugzilla.redhat.com/show_bug.cgi?id=1069840>
18:54 dbruhn JoeJulian, did you ever finish your split brain cleanup script you were working on?
18:55 JoeJulian dbruhn: That's what cp0k was asking about. Maybe today...
18:56 dbruhn I have that bash script I was working on at a state that is lists the files, lets you choose. But it's honestly a mess
18:56 dbruhn and does things a little differently as I copy the good file out to a temp location, and nuke it from the file system
18:56 JoeJulian dbruhn: I have an elegant solution, just trying to lower the barrier to entry for it. Right now it requires understanding the translator graph.
18:57 dbruhn ahh damn
18:59 kmai007 graph now that is a word i see alot in the gluster logs
18:59 kmai007 graph 0, graph 1
18:59 kmai007 i have no idea what graph its referring to
18:59 JoeJulian The translators, all assembled together, are a graph.
18:59 kmai007 thank you for that explaination
19:00 JoeJulian Still no idea what that's referring to though.
19:00 kmai007 at least we are in agreement
19:01 kmai007 this is insane but, i honestly like cleaning up split-brain files, well not daily, but last week it was perfect in a cozy winter day LOL
19:03 dbruhn kmai007, how many are you dealing with? lol
19:04 kmai007 last week the max was 1023
19:05 kmai007 but in that it was repeatative gfid's
19:05 kmai007 with differ timestamps
19:05 kmai007 i cleaned up lets say 13 unique ones, and the rest disappeared
19:06 cfeller joined #gluster
19:06 dbruhn agh, yeah that is easier to deal with. We have been battling with some rebalance ops that have caused something like 2000 unique ones every time the issue comes up.
19:06 kmai007 yeh i don't know how things get in that state, when it stays "connected"
19:07 kmai007 and when it loggs disconnected, i don't know how or why
19:07 kmai007 from the FUSE
19:07 kmai007 yet i can traverse the filesystem
19:07 cjanbanan joined #gluster
19:07 kmai007 my thought is its being bombarded so much it gives up and says, out to lunch
19:08 JoeJulian Have you messed with ping-timeout?
19:08 kmai007 i left it as default
19:08 kmai007 then increased it to 60 seconds
19:08 kmai007 when i hear expensive operation, i figured lower is not better
19:09 kmai007 oh so ping-timeout, what exactly does it do in operation?
19:09 kmai007 does it drop writing/reading from that particular brick ?
19:09 kmai007 additionally i don't get a specific brick IP, that iv'e seen before
19:10 kmai007 i get more of replicate-1 disconnecting after 42 seconds
19:10 kmai007 super strange, because gluster peer status and volume status, and volume info show it is active alive, and full of wonderful life.....
19:10 dbruhn that is strange
19:11 dbruhn do you have something on your network that could be momentarily disrupting it? Maybe a monitoring solution or something along those lines?
19:11 pdrakeweb joined #gluster
19:11 kmai007 but *knocking on wood* after i've increased it to 60 seconds, its been pretty stable
19:12 kmai007 yeh i thought maybe 42 is too early to disconnect so increased it to 60
19:13 dbruhn funny, more often than not people are trying to figure out how to reduce that time for vm's so they don't lock and turn read-only
19:15 aquagreen1 joined #gluster
19:16 pdrakeweb joined #gluster
19:16 cjanbanan joined #gluster
19:22 elyograg I continue to have high CPU problems with a second gluster volume even after upgrading to 3.4.2.  A new proiblem has surfaced since the upgrade too -- I cannot get 'gluser volume heal XXXX info' to respond on that other volume.  works fine on the first volume.  Also, when I run that info command, it hangs the commandline and no other gluster volume commands can be run until I completely kill gluster and restart it on the server where I ran the comm
19:29 elyograg current load on one server is about 28.  I saw it get up to 38.  one quad-core processor.
19:30 elyograg unlike when it was on 3.3.1, operations at the other end of an NFS mount do see to continue working, just slowly.
19:30 cjanbanan joined #gluster
19:33 elyograg the really bizarre thing -- everything on the other volume (the one that came first) works with ZERO problems.  I've got some gfids that show up on it's heal info report after the upgrade, but everything else is good.
19:33 elyograg both volumes are accessed via NFS through a gluster peer that has no bricks.
19:37 elyograg the only remaining difference between the four machines whose bricks make up the first volume and the two machines that make up the new volume is that the 4TB disks installed are SATA rather than SAS.  Both kinds of drives are western digital Raid Edition enterprise drives.
19:50 kmai007 high cpu, are you feeling that through your clients as well?
19:51 tdasilva joined #gluster
19:51 elyograg yes.  when it gets this bad (load now up to 42 on one machine) evertyhing grinds to a halt and I end up completely shutting down gluster on both servers, then restarting it.  I may actually reboot those servers this time.
19:52 MacWinne_ joined #gluster
19:52 kmai007 to build an understanding its a particular volume? or all volumes on that brick ?
19:52 elyograg a particular volume.
19:54 elyograg fpaste.org/80341/35790713/ mdfs is the volume that works with no problems.  mdfs2 is the second volume that has had high cpu problems since we put it in.
19:54 elyograg stupid firefox.  leaves out the damn http://
19:55 elyograg http://fpaste.org/80341/35790713/
19:55 jag3773 joined #gluster
19:55 * elyograg pokes glusterbot.
19:56 kmai007 top shows you that its glusterd ?
19:56 kmai007 that has the most load
19:56 kmai007 is there any backup software running on that server right now?
19:57 kmai007 are all the mounts on the bricks mounted?
19:57 kmai007 does dmesg show you anything useful?
19:58 elyograg no backup software.  it is glusterfsd processes corresponding to bricks that have high CPU load.
19:58 dewey Oh mighty gods of gluster configuration:   when I peer probe <otherhost> my peer status shows <otherhost> as peer.  On the other host, it shows <IP of first host>.  I'm thinking it should show <firsthost name>.  What am I doing wrong?
19:59 elyograg all the bricks are mounted.  the volume won't start if they're not - I use a subdirectory of the mountpoint for the brick path.
19:59 elyograg dewey: peer the firt probe by name from the one you just added.
19:59 elyograg probe the first peer.  wow, not doing well today.
19:59 kmai007 elyograg: if you can afford it, reboot that brick
20:00 kmai007 its hard to understand how it went haywire
20:00 kmai007 did you upgrade to 3.4.2 while glusterd was down?
20:00 dewey elyograg -- now I have the same situation in reverse.
20:01 elyograg I shut down all gluster processes before upgrading.  eight servers were upgraded last night.  two servers acting as a redundant NFS access point, four servers with bricks for the first volume, and two servers with bricks for the second volume.
20:02 elyograg rebooted the machine with high load.  now the other machine with bricks for that volume is showing escalating load.
20:03 cjanbanan joined #gluster
20:03 elyograg typically the problem will not go away until I have stopped all gluster processes and restarted glusterd on both servers.
20:04 elyograg stopped on both at the same time, that is.
20:04 elyograg rebooting the other server before the first comes back up.
20:05 elyograg once we get this into production, we cannot just reboot this. we can't put it into production when it's like this, though.  and we're running out of space.
20:06 kmai007 understood,
20:06 kmai007 any indicatio in the /var/log/glusterfs/glustershd.log ?
20:06 kmai007 that its actually doing some productive work?
20:07 elyograg we wouldn't have made a second volume at all, except that we ran into horrible problems during our first add/rebalance attempt.  after looking at the problem and discovering that rebalancing would become unworkable as we grew to hundreds of terabytes, we decided to put the next 40TB (80TB raw) into its own volume.
20:07 elyograg 3.3.1 has REALLY BAD BUGS with rebalance.
20:08 dbruhn +1 I've been bit by the rebalance bug too
20:08 dbruhn memory leaks, and it will take bricks offline
20:09 elyograg if any brick becomes unresponsive during the rebalance for more than the timeout value (the 42 second one, still don't know why that happened), you'll have data loss, permission changes, and other problems.
20:09 dbruhn Does it annoy anyone else that some open source projects are starting to release vagrant stuff and containers without accompanying build information?
20:10 ctria joined #gluster
20:10 elyograg after reboot, the high load is back.
20:10 kmai007 eww....
20:10 dbruhn any self heals going on?
20:11 elyograg not that I know of.
20:12 kmai007 what if you left that brick down?
20:12 kmai007 how is performance on that volume?
20:12 elyograg there are eight bricks per machine.  which one?
20:12 Philambdo joined #gluster
20:13 cjanbanan joined #gluster
20:13 nightwalk joined #gluster
20:15 B21956 joined #gluster
20:26 sputnik1_ joined #gluster
20:28 primechuck Is the stripe translator with distribute+replicate still considered scary?  Or have people been using it in some big installs with big files yet?
20:41 JoeJulian I haven't heard of more than one or two people.
20:46 XpineX joined #gluster
20:48 ndevos joined #gluster
20:48 cjanbanan joined #gluster
20:52 rpowell joined #gluster
21:01 theron joined #gluster
21:03 cjanbanan joined #gluster
21:13 dewey and my next interesting problem:  One my (attempted) new gluster node, all volumes are showing up with "listen-port=0".  this makes gluster not want to start.
21:14 dewey I've tried volume sync  -- in fact, my work-around to restart is to remove the vol files and resync, but I'd rather like my node to survive a reboot.
21:16 elyograg the impression I'm getting with gluster is that it doesn't deal well with accessing more than one volume through a single NFS point.
21:16 elyograg an overriding impression is that gluster isn't going to fill our needs at all in the long term.  I'd love to be wrong about that.
21:17 dewey elyograg -- you mean mounting multiple volumes through a single node?
21:18 elyograg yes.  i cannot handle the overhead of having three IP addresses taken up in each of my networks for every gluster volume.
21:18 kl4m joined #gluster
21:19 dewey Hmmm.  Interesting.  I'm using the gluster FUSE client.
21:19 dewey I did try NFS at one point and had problems that I *thought* was unrelated to Gluster but perhaps I should reconsider.
21:21 elyograg The storage must be accessible to multiple IP networks, which are different zones off our firewalls.  Storage traggic is too much for going *through* the firewalls.
21:22 kl4m_ joined #gluster
21:22 dewey I hear you on that.  I'm doing something similar only with SAMBA (and therefore and external system) rather than NFS.
21:23 dewey I did discover that SAMBA locks and Gluster file locks don't play nice together, but after resolving that it's been fine (on a single node -- as I've been waiting for a replacement 2nd node)
21:23 elyograg Ultimately I expect to be using samba and nfs.
21:24 rwheeler joined #gluster
21:24 dewey ahhh.  I had one interesting problem under SAMBA:  I had a SQL database backup process writing large and small backups to SAMBA which was backed by Gluster -- we kept getting file lock issues.  turning off native locking  (posix locking = no) on the share allowed it to work.
21:25 dewey It's *only* a single share that is used by Litespeed SQL server backups.  Other shares being accessed through both interactive users and automatic processes have had no issues.
21:26 dewey But it does mean that I can't effectively load-balance using a 2nd SAMBA head on that share.
21:28 andreask joined #gluster
21:35 kl4m joined #gluster
21:35 dewey This is quite a mess:  I have a (failed) brick in a volume that cannot be removed.
21:36 cjanbanan joined #gluster
21:42 ProT-0-TypE joined #gluster
21:42 cjanbanan joined #gluster
21:53 lman482 joined #gluster
21:56 lman482 Trying to get ecryptfs working… not able to write to files .. getting this in logs: kernel: Either the lower file is not in a valid eCryptfs format, or the key could not be retrieved. Plaintext passthrough mode i
21:56 lman482 s not enabled; returning -EIO
22:03 cjanbanan joined #gluster
22:04 jag3773 joined #gluster
22:10 cjh973 anyone have an example of using the glusterfs api in a c++ program?
22:10 cjh973 i'm having trouble getting it to compile
22:17 nage joined #gluster
22:17 nage joined #gluster
22:24 semiosis cjh973: well known examples of libgfapi use are qemu & samba... idk if either of those are C++ though
22:25 cjh973 semiosis: ok i'll check those out
22:26 cjh973 semiosis: looks like qemu is C
22:27 semiosis isnt c code also c++ code/
22:27 semiosis ?
22:27 cjh973 i'm not sure.  i'm a beginner :)
22:28 semiosis why c++?
22:28 cjh973 i donno.  it's easier for me because i come from java
22:28 JonnyNomad joined #gluster
22:28 semiosis you know, i am writing a ,,(java) connector for libgfapi
22:28 glusterbot https://github.com/semiosis/libgfapi-jni & https://github.com/semiosis/glusterfs-java-filesystem
22:28 dbruhn semiosis, has a java interface to gluster
22:28 dbruhn lol
22:29 semiosis the first is a low level jni library, the second is a high level filesystem provider implementation
22:29 cjh973 sweet
22:29 semiosis what is your end goal?
22:29 cjh973 i want to create an S3 interface to gluster
22:30 cjh973 with a little embedded civetweb server
22:30 semiosis well if you're using java, i think jetty is the most popular embedded web server
22:30 cjh973 it is quite popular
22:31 cjh973 semiosis: maybe i would be better served to drop down to C for this
22:32 semiosis idk
22:32 semiosis what i would do is find an s3 compatible web service already written in java
22:32 cjh973 do you know of any other way to layer s3 onto gluster?
22:32 cjh973 there's not many s3 servers out there that i can find
22:33 cjanbanan joined #gluster
22:33 semiosis 10 seconds of googling found this... https://code.google.com/p/littles3/
22:33 dbruhn you could put swift on top of it
22:33 glusterbot Title: littles3 - Server implementation of Amazon S3 API - Google Project Hosting (at code.google.com)
22:33 cjh973 yeah i saw that also.  i could give that a shot
22:33 semiosis which is almost 5 years old :(
22:33 semiosis eek
22:33 cjh973 yeah
22:33 cjh973 it's not maintained
22:34 semiosis anyway, what i was going to say earlier...
22:34 semiosis find or write an S3 web service in java and use the java7 nio.2 filesystem api for the file io
22:34 jiqiren i'm working on upgrading from 3.3.0 to 3.4.2, i'm having trouble getting my volume to heal
22:34 semiosis then just drop in glusterfs-java-filesystem :)
22:34 cjh973 that's not a bad idea :)
22:34 semiosis cjh973: if i were starting out from scratch i would probably use dropwizard
22:35 cjh973 what is that?
22:35 semiosis @lucky dropwizard
22:35 glusterbot semiosis: http://dropwizard.codahale.com/
22:35 semiosis a java rest api microframework
22:35 cjh973 heh interesting
22:36 semiosis although s3 uses xml, dropwizard is primarily concerned with json, that shouldnt matter too much
22:36 cjh973 yuck xml
22:36 semiosis well you want s3
22:36 cjh973 yeah
22:36 semiosis if you dont need s3 compat.....
22:36 cjh973 i didn't realize it was using xml
22:37 cjh973 from looking at the spec it looked like it was just servicing get/put requests
22:38 semiosis i guess it really just depends on how much of the API you want to implement
22:38 cjh973 semiosis: you're right.  i see the xml in there now
22:38 cjh973 yeah
22:38 cjh973 i wonder why they went for that instead of json.  xml is nasty
22:43 semiosis but wait, it gets even worse... http://stackoverflow.com/a/9164234
22:44 semiosis glusterbot: ping
22:44 glusterbot pong
22:45 cjh973 damn
22:45 cjh973 xml it is then
22:53 plarsen joined #gluster
22:54 nage joined #gluster
22:54 badone joined #gluster
23:01 cjanbanan joined #gluster
23:25 RicardoSSP joined #gluster
23:25 diegows joined #gluster
23:35 tdasilva left #gluster
23:42 cjanbanan joined #gluster
23:45 theron joined #gluster
23:52 glusterbot New news from newglusterbugs: [Bug 991035] ACL mask is calculated incorrectly <https://bugzilla.redhat.com/show_bug.cgi?id=991035>
23:58 dbruhn left #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary