Camelia, the Perl 6 bug

IRC log for #gluster, 2013-02-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 andrei JoeJulian: when you stop the glusterfs-server service, should it not kill all the glusterfs related processes?
00:06 JoeJulian No, just glusterd
00:06 andrei i've stopped it, however, I can still see several processes when I do ps aux
00:06 JoeJulian If you're clients have unmounted the volume, you can safely killall glusterfsd
00:06 andrei so, it is notmal to see glusterfsd processes on the server?
00:07 JoeJulian and killall glusterfs for that matter
00:07 JoeJulian It is.
00:07 JoeJulian To stop those you would either cheat like I'm suggesting, or stop the volumes through the cli.
00:08 jjnash It's possible to downgrade using apt-get, but it's messy and definitely not straight-forward
00:08 JoeJulian and I would suspect that the deb would have to be available in the ppa, right?
00:08 jjnash oh, crud. my client withheld a bunch of messages again. disregard ^
00:09 JoeJulian hehe, I hate when I'm scrolled up.
00:10 jjnash This particular issue is where the client doesn't switch back to autoscroll even when I return it to the bottom of the output
00:11 jjnash To make matters worse, it's seemingly inconsistent about when it chooses to versus when it chooses not to
00:11 JoeJulian What client are you using?
00:11 andrei JoeJulian: after doing the upgrade, the port 24008 has disappeared
00:11 jjnash pidgin
00:11 jjnash (don't laugh)
00:11 andrei and it's not listening
00:12 JoeJulian I used to until they told me that I was doing it wrong. Now I'm doing the same thing with XChat and it works fine. The Pidgin devs are a-holes.
00:13 JoeJulian andrei: got that log file for me?
00:13 andrei one sec
00:13 andrei and I can also see that the nfs shares are not presented
00:14 andrei showmount -e doesn't show anything
00:18 andrei here you go
00:18 andrei http://pastebin.com/Y0gQqbZ7
00:18 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
00:18 andrei sorry bot )))
00:18 JoeJulian Yeah, repaste there please. Not only the adds but too much screen real estate is taken up by those add blocks.
00:18 JoeJulian s/add/ad/g
00:19 glusterbot JoeJulian: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
00:19 JoeJulian glusterbot: meh
00:19 glusterbot JoeJulian: I'm not happy about it either
00:20 andrei http://dpaste.org/6aaQI/
00:20 glusterbot Title: dpaste.de: Snippet #218450 (at dpaste.org)
00:20 andrei is this better?
00:20 JoeJulian thanks
00:20 andrei thank you!
00:21 andrei i've upgraded to 3.3.1 from debian ppa without uninstalling it and reinstalling it first
00:24 bala joined #gluster
00:25 melanor9 joined #gluster
00:25 bala joined #gluster
00:27 JoeJulian Well that's interesting...
00:27 andrei )))
00:28 JoeJulian Are you saying that it's working now?
00:28 andrei that phrase usually worries me!
00:28 andrei nope
00:28 andrei it's not working
00:28 andrei i can't mount the glusterfs volume from the client
00:28 JoeJulian Well your log shows  [2013-02-04 23:55:04.699386] E [rdma.c:4512:gf_rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 192.168.168.201:1022) after handshake is complete
00:28 andrei th emount just hangs
00:29 JoeJulian What's interesting about that is that gf_rdma_event_handler isn't at line 4512.
00:29 andrei JoeJulian: life is great when people know by heart the calls and the lines in code that they are in )))
00:29 JoeJulian Oh, nevermind...
00:30 JoeJulian Oh, I don't have that great of a knowledge. But it's all right there....
00:30 andrei what i've noticed is that i've not killed all glusterfs processes before doing the version upgrade
00:30 andrei perhaps I should kill everything and restart gluster sever?
00:30 JoeJulian I was opening it to read so I could try to make heads or tails of what that error actually means.
00:30 JoeJulian yes
00:30 andrei i'll do that now
00:30 andrei one moment
00:31 JoeJulian Ah, that was a 3.3.0 error. You didn't truncate.
00:31 andrei I will do that now
00:32 andrei however, it did work with 3.3.0
00:32 andrei )))
00:33 JoeJulian I understand. But in trying to debug things, I like to be able to look at the source. When it says an error occurred at a specific line, but that line in the version I think we're talking about it blank, that makes it a bit harder. :)
00:34 johndescs_ joined #gluster
00:34 andrei sorry mate
00:34 andrei my bad
00:36 andrei here is a clean log
00:36 andrei http://dpaste.org/ewibn/
00:36 glusterbot Title: dpaste.de: Snippet #218452 (at dpaste.org)
00:37 JoeJulian netstat -tlnp | grep 24008
00:38 andrei i can see that the client is trying to reach port 24008
00:38 andrei 00:37:22.286785 IP 192.168.168.1.1023 > 192.168.168.200.24007: Flags [.], ack 4273026981, win 135, length 0
00:38 andrei 00:37:22.286813 IP 192.168.168.200.24007 > 192.168.168.1.1023: Flags [.], ack 1, win 86, length 0
00:38 andrei 00:37:22.735153 IP 192.168.168.200.24007 > 192.168.168.1.1023: Flags [.], ack 1, win 86, length 0
00:38 andrei 00:37:22.735237 IP 192.168.168.1.1023 > 192.168.168.200.24007: Flags [.], ack 1, win 135, length 0
00:38 andrei 00:37:23.089007 IP 192.168.168.1.1022 > 192.168.168.200.24008: Flags [S], seq 394800873, win 65480, options [mss 65480,nop,wscale 9], length 0
00:38 andrei 00:37:23.089036 IP 192.168.168.200.24008 > 192.168.168.1.1022: Flags [R.], seq 0, ack 394800874, win 0, length 0
00:38 andrei 00:37:26.097661 IP 192.168.168.1.1022 > 192.168.168.200.24008: Flags [S], seq 441811107, win 65480, options [mss 65480,nop,wscale 9], length 0
00:38 andrei 00:37:26.097690 IP 192.168.168.200.24008 > 192.168.168.1.1022: Flags [R.], seq 0, ack 47010235, win 0, length 0
00:38 andrei but that port is not open
00:38 andrei was kicked by glusterbot: message flood detected
00:38 andrei joined #gluster
00:38 andrei sorry
00:39 JoeJulian I think you're the first time that's worked...
00:39 andrei )))
00:39 andrei the netstat stuff
00:39 andrei http://dpaste.org/hY0YG/
00:39 JoeJulian Ok, why do you think 24008 isn't connecting?
00:39 glusterbot Title: dpaste.de: Snippet #218453 (at dpaste.org)
00:40 JoeJulian Because it's ack'ing packets.
00:40 andrei i do not see 24008 being open
00:40 andrei JoeJulian: the IP ending with .1 is the client
00:41 andrei and .200 is the server
00:41 andrei however
00:41 andrei that's strange
00:41 JoeJulian yes.
00:41 andrei you are right, 24008 is answering back
00:41 andrei does it temporarily open that port?
00:41 andrei coz in 3.3.0 it was open all the time
00:42 JoeJulian Maybe it's an rdma thing...
00:42 andrei JoeJulian: the netstat doens't show rdma stuff
00:42 andrei it's not an IP
00:42 JoeJulian mmkay
00:42 JoeJulian Then I guess that would make sense.
00:43 JoeJulian So... on to the client.
00:43 andrei gluster should start communication over IP and negotiate the rdma connections
00:43 JoeJulian You say the client's not connecting?
00:43 andrei coz I remember when 24008 was open, If i telnet to it it would give me a bunch of stuff for rdma
00:43 andrei JoeJulian: that's right, the client is not connecting
00:44 JoeJulian I'm just going to assume (for the moment) that the server's working perfectly. Let's clean a client log, try it, and see what we have.
00:44 andrei okay
00:44 andrei all yours
00:44 raven-np joined #gluster
00:45 andrei what should i do?
00:46 andrei clean the logs and try to mount?
00:46 JoeJulian Please
00:50 andrei one sec
00:50 andrei almost there
00:51 andrei http://dpaste.org/BeWOp/
00:51 glusterbot Title: dpaste.de: Snippet #218455 (at dpaste.org)
00:51 JoeJulian Do you have any ipv6 in this?
00:52 andrei i am not using ipv6
00:52 andrei but the server might have it enabled by default
00:52 andrei i've not switched it off
00:52 andrei why do you ask?
00:53 JoeJulian Some other bug I encountered a while back.
00:53 JoeJulian Ok, the fact that you get, "Given volfile:..." means that it's connecting to glusterd.
00:53 andrei centos has an ipv6 enabled
00:54 andrei yeah, that's right
00:54 andrei but the next thing after getting the file is connection refused
00:55 JoeJulian Right... it's refusing connection to the glusterfsd (brick server). Let's check the brick log on arh-ibstorage-ib for this volume.
00:55 JoeJulian What's an rdma name resolution look like? Is it still an IP address?
00:56 andrei basically this is what's happening
00:56 andrei at least from the tcpdump point of view
00:56 andrei when the client is mounting volume
00:56 andrei i can see that the client is connecting to port 24008 amongst other ports
00:56 andrei and that port is not listening
00:56 andrei [root@arh-cloud1 glusterfs]# telnet arh-ibstorage-ib 24008
00:56 andrei Trying 192.168.168.200...
00:56 andrei telnet: connect to address 192.168.168.200: Connection refused
00:57 andrei this is where the Connection refused part is showing in the client logs
00:57 andrei so we need to figure out when glusterfs is not opening this port
00:57 andrei to answer your question about rdma/Ip
00:58 andrei gluster should first connect via IP to port 24008 where it will simply get a few text lines
00:58 andrei these lines contain the rdma stuff that is required to connect
00:58 andrei like what LID to use and what keys to use, etc
00:58 andrei this information is used to establish the actual rdma data connection
00:59 andrei as port 24008 is not opened, the client doesn't know what to do next and how to create the rdma connection
00:59 andrei [2013-02-05 00:54:31.203813] E [rdma.c:4604:tcp_connect_finish] 0-cloudstack-primary-client-1: tcp connect to  failed (Connection refused)
00:59 andrei this entry in the log corresponds to the client trying to connect to port 24008
00:59 andrei and you get Connection refused, just like when I try to telnet to that port
01:00 andrei I am sure that 3.3.0 gluster had this port open as I've checked for it in the past
01:00 andrei perhaps there is a stupid configuration issue which can be fixed by one line of config entry
01:00 andrei as it happens sometimes
01:01 JoeJulian Try reversing rdma and socket in /etc/glusterfs/glusterd.vol, ie "option transport-type rdma,socket" I've seen something like that before but that should be fixed.
01:02 andrei JoeJulian: looking at the open ports, it seems that the rdma info is actually given over port 24009
01:02 JoeJulian ... and nothing to do with that changed between 3.3.0 and 3.3.1 afaict.
01:02 andrei telnet localhost 24009
01:02 andrei Trying 127.0.0.1...
01:02 andrei Connected to localhost.
01:02 andrei Escape character is '^]'.
01:02 andrei QP1:RECV_BLKSIZE=00000800:SEND_BLKSIZE=00000800
01:02 andrei QP1:LID=0001:QPN=3e004b:PSN=000000
01:02 andrei DONE
01:02 JoeJulian That's the brick.
01:02 andrei so, probably port 24008 is used for something else
01:03 andrei this part - QP1:LID=0001:QPN=3e004b:PSN=000000
01:03 andrei this is infiniband / rdma stuff
01:03 andrei for sure
01:03 JoeJulian If you really want to be perplexed, re-read the error message carefully.
01:04 JoeJulian "tcp connect to  failed"
01:04 andrei could we have the case of things not listening on the right ports?
01:04 andrei yeah, i've seen this
01:04 andrei )))
01:04 JoeJulian The source reads, "tcp connect to %s failed (%s)"
01:04 andrei ah
01:05 JoeJulian meaning that this->peerinfo.identifier is null
01:05 andrei okay
01:05 JoeJulian But that may be a red herring. I've been led down that path before.
01:05 andrei )))
01:06 JoeJulian Let's try the client with loglevel=DEBUG
01:06 andrei could you please let me know how to enable debug mode?
01:07 andrei where should I specify this option?
01:08 JoeJulian mount -t glusterfs -o log-level=DEBUG
01:08 andrei thanks
01:08 JoeJulian m0zes: are you awake yet?
01:09 JoeJulian andrei: if nothing pops out from this, I'm going to ask for packet captures to look at with wireshark
01:09 mkultras joined #gluster
01:09 andrei sure, no probs
01:10 mkultras hey is 3.2.5 client able to mount a 3.0.2 server ?
01:10 mkultras or not rpc compatible?
01:11 m0zes JoeJulian: yes?
01:11 JoeJulian mkultras: No. Minor version changes broke compatibility.
01:11 JoeJulian m0zes: You're our resident ib expert... :)
01:12 m0zes hah! i've not gotten it working on 3.3
01:12 JoeJulian Ok, go back to sleep then. ;)
01:12 andrei http://dpaste.org/meOGI/
01:12 glusterbot Title: dpaste.de: Snippet #218456 (at dpaste.org)
01:12 andrei the debug logs
01:13 m0zes it worked for 3.2. not sure why it doesn't work in 3.3
01:13 andrei m0zes: it seems to work with 3.3.0
01:13 andrei it's not working in 3.3.1
01:14 plarsen joined #gluster
01:14 andrei JoeJulian: look at this: [2013-02-05 01:13:52.298539] D [name.c:208:af_inet_client_get_remote_sockaddr] 0-cloudstack-primary-client-0: option remote-port missing in volume cloudstack-primary-client-0. Defaulting to 24008
01:14 andrei for some reason it's not getting the right port from the server
01:15 andrei it should be 24009
01:15 m0zes and before i upgraded to 3.3.1 from 3.2.7 i was told it didn't work in 3.3.0 but did in 3.3.1
01:15 andrei but it's getting 24008 as a default
01:15 andrei m0zes: I am the leaving proof otherwise )))
01:15 JoeJulian andrei: Good catch... I wonder if we can force that.
01:15 andrei me too
01:16 andrei leaving = living )))
01:18 andrei bricks/arh-ibstorage-ib:-zfs-mirror-pri​mary-ib-kvm-cluster1:listen-port=24010
01:18 andrei bricks/arh-ibstorage-ib:-zfs-mirror-pri​mary-ib-kvm-cluster1:rdma.listen-port=0
01:18 andrei this is what i've found in /var/lib/glusterd folder
01:19 m0zes something on the wiki would be wonderful if you figure it out :-)
01:19 andrei the rdma.listen-port is shown as 0
01:19 andrei not sure if this is becuase rdma works differently from ip and doesn't use ports as such (from what I understand)
01:19 andrei or this is becuase of a bug in 3.3.1
01:20 JoeJulian I'm leaning toward bug... Oh, let's see if name.c changed...
01:21 JoeJulian One change...
01:22 andrei is it related to this issue?
01:22 JoeJulian Doesn't look like that should do it, but it may be somewhere else...
01:22 JoeJulian Ok, let's edit volfiles...
01:23 andrei where are they located?
01:23 andrei here: /var/lib/glusterd/vols/cloudstack-primary/bricks
01:23 andrei ?
01:24 andrei or in /etc/glusterfs ?
01:24 JoeJulian edit /var/lib/glusterd/vols/cloudstack-primary-fuse.vol
01:25 JoeJulian For each type "protocol/client" section, add "option remote-port %s" where %s is the port that the brick is listening on.
01:25 andrei do you mean this one: trusted-cloudstack-primary-fuse.vol?
01:25 JoeJulian yes
01:25 JoeJulian er,
01:25 andrei and the port should be 24009 and not 24008, right?
01:26 semiosis @ports
01:26 glusterbot semiosis: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
01:26 melanor9 joined #gluster
01:26 JoeJulian netstat -tlnp | grep glusterfsd
01:26 JoeJulian Make 'em match
01:26 andrei will do that now
01:26 JoeJulian semiosis: Pretty sure we've found a bug in port resolution for rdma bricks in 3.3.1
01:26 semiosis been keeping an eye on this conv
01:27 mkultras hey i just downloaded the gluster 3.0 from gitup did ./configure make make install and tried to mount and got undefined symbol: set_global_ctx_ptr
01:27 mkultras on ubuntu12
01:27 semiosis mkultras: gluster 3.0?  ,,(latest) is 3.3.1
01:27 glusterbot mkultras: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
01:27 mkultras ohya i'm trying to mount a gluster server that is 3.0
01:27 semiosis :(
01:28 andrei netstat | grep glusterfsd gives me 3 ports
01:28 andrei tcp        0      0 0.0.0.0:24009           0.0.0.0:*               LISTEN      24321/glusterfsd
01:28 andrei tcp        0      0 0.0.0.0:24010           0.0.0.0:*               LISTEN      24315/glusterfsd
01:28 andrei tcp        0      0 0.0.0.0:24011           0.0.0.0:*               LISTEN      24309/glusterfsd
01:28 JoeJulian grep those from ps. You should be able to read the line to figure out which brick is which.
01:28 mkultras oh this might be from me installing and removing gluster with apt-get first i think
01:29 sjoeboo joined #gluster
01:29 * m0zes still needs to file a bug against 3.3.1 for mismatched hw kicking peers out of the pool
01:29 glusterbot http://goo.gl/UUuCq
01:30 andrei oh, i see
01:30 andrei for each volume it opens a new port
01:30 andrei thanks for the tip )
01:30 JoeJulian For each brick
01:30 JoeJulian You /can/ have multiple bricks per volume on one server.
01:31 andrei yeah, i've done that
01:31 andrei 2 additions to that file
01:31 andrei as i've got 2 bricks
01:32 andrei should I now restart glusterfs server?
01:32 JoeJulian No, just try mounting the client
01:32 andrei okay
01:32 andrei one sec
01:33 andrei does it apply changes on the fly?
01:33 JoeJulian I have a fuse.vol file without the "trusted-" prefix. So if that doesn't work, check for that one too.
01:33 JoeJulian It's the client configuration, so it'll be read on client startup.
01:34 andrei JoeJulian, oh, sorry
01:34 andrei i've eddited the server one
01:34 andrei should I revert it back and try editing the client side file?
01:35 JoeJulian let me scroll back and see if I misread something...
01:36 JoeJulian No, that looks right. Since you specify this server when mounting the volume, the client will read /var/lib/glusterd/vols/cloudstack-primary-fuse.vol (maybe /var/lib/glusterd/vols/trusted​-cloudstack-primary-fuse.vol?) for it's configuration.
01:36 JoeJulian from this server
01:36 andrei on the client the folder /var/lib/glusterd/ is empty
01:36 JoeJulian right
01:37 JoeJulian There's ways of doing it where we copy the .vol file to the client, but let's just stick with this for now. If it works, great. If not, we'll copy it to the clients.
01:38 andrei it did't work - according to the log it got the .vol file, but the file is still the old one
01:38 andrei without my changes
01:39 JoeJulian Probably not the "trusted-" one then.
01:39 andrei however, i've not restarted the server
01:39 JoeJulian If you restart the server, this will likely get overwritten.
01:40 andrei here is what i've got:
01:40 andrei -rw------- 1 root root 1676 Feb  4 01:13 cloudstack-primary.arh-ibstorage2-ib.z​fs-mirror-primary-ib-kvm-cluster1.vol
01:40 andrei -rw------- 1 root root 1676 Feb  4 01:13 cloudstack-primary.arh-ibstorage-ib.z​fs-mirror-primary-ib-kvm-cluster1.vol
01:40 andrei -rw------- 1 root root 1633 Feb  4 01:13 cloudstack-primary-fuse.vol
01:40 glusterbot andrei: You've given me 5 invalid commands within the last minute; I'm now ignoring you for 10 minutes.
01:40 JoeJulian what are you talking about glusterbot?
01:40 andrei should I edit them?
01:41 JoeJulian just cloudstack-primary-fuse.vol
01:41 JoeJulian Oh, that's why... I missed the directory in that path...
01:42 JoeJulian /var/lib/glusterd/vols/cloudstack-p​rimary/cloudstack-primary-fuse.vol
01:44 andrei okay, a different set of errors this time
01:44 andrei but still no mount
01:45 JoeJulian Do you have any firewall?
01:46 andrei http://dpaste.org/z5icS/
01:47 andrei not on the server side
01:47 andrei allow all
01:47 JoeJulian ok
01:51 JoeJulian ok! so arh-ibstorage-ib worked, but arh-ibstorage2-ib didn't.
01:51 JoeJulian Check arh-ibstorage2-ib processes and make sure we targeted the right listener.
01:51 andrei will do, one moment
01:51 andrei did arh-ibstorage worked?
01:52 JoeJulian [2013-02-05 01:42:34.425253] I [client-handshake.c:1433:client_setvolume_cbk] 0-cloudstack-primary-client-0: Connected to 192.168.168.200:24010, attached to remote volume '/zfs-mirror/primary-ib-kvm-cluster1'.
01:52 JoeJulian a2: Are you lurking?
01:52 andrei but I still can't get the mount point
01:52 andrei i mean it hangs
01:52 andrei and doesn't give me any info
01:53 andrei like df and ls don't work
01:53 andrei the arh-ibstorage2-ib is offline ( i mean the glusterfs is off)
01:53 andrei i will start it
01:54 JoeJulian I presume the df/ls tests were after the end of the log you posted?
01:54 andrei okay
01:55 andrei i've started gluster server on the second storage server
01:55 JoeJulian what time is it in Bangalore anyway....
01:55 andrei and will know try to mount once again
01:55 JoeJulian Ooh, raghaven1rabhat should be in any time...
02:00 andrei JoeJulian: okay, here is what i've got so far
02:00 andrei after starting glusterfs on the second server the mount command on the client side finished
02:00 andrei and I can see df -h info on the mountpoint
02:01 andrei when I tried to do ls -la /mountpoint it got stuck
02:01 andrei after that i've killed the mount processes and done umount -fl /mountpoint
02:01 JoeJulian Oooh, that kind of stuck.
02:01 andrei following that my ls -la command gave me this:
02:02 andrei ls: cannot access /tmp/testing/9315ff87-873c-4ba7-9959-abea9d8ab1df: No such file or directory
02:02 andrei ls: cannot access /tmp/testing/00f48973-6e7c-4a5c-8bd2-b960408a71b8: No such file or directory
02:02 andrei ls: cannot access /tmp/testing/e6a71c75-5084-4a5d-bafc-5b32b34ff037: No such file or directory
02:02 andrei and a lot more like these
02:02 JoeJulian Which would make sense since you killed the mount.
02:02 JoeJulian Let's see the client log.
02:02 andrei however, this might be because both servers are not yet in sink?
02:02 andrei as the second server hasn't finished the heal process
02:03 andrei so the data on both servers are different
02:03 JoeJulian possible. I've filed a bug about something similar.
02:03 andrei i've only managed to get around 150gb
02:04 JoeJulian The client should show "background self-heal" messages
02:04 andrei i am not sure that the heal is running now
02:04 copec joined #gluster
02:05 JoeJulian I'm not either... We're kind-of in force-it-to-work territory. That client should have started healing the files that got touched when you did the ls.
02:05 JoeJulian In fact... I rather doubt the self-heal daemon's working yet. It has it's own vol file.
02:06 JoeJulian To try to fix it, before you kill it, make a note of the entire command line. We're going to want to run it that way instead of through glusterd.
02:06 andrei ah!
02:06 andrei i am looking at the client logs now
02:06 andrei and here is what I've got:
02:06 JoeJulian brb... pee break!
02:07 andrei http://dpaste.org/cOiGq/
02:07 glusterbot Title: dpaste.de: Snippet #218461 (at dpaste.org)
02:07 andrei a bunch of entries like that
02:09 JoeJulian Yep, that's self-heals.
02:09 shapemaker joined #gluster
02:09 JoeJulian I think the hang was because the heal queue was full.
02:10 JoeJulian I don't like that it does that, I've filed a bug for that too. Need to better document it though.
02:10 andrei so what should I do next?
02:11 andrei to heal both servers?
02:11 andrei and get the mountpoint to respond
02:11 JoeJulian How long should the full self-heal take?
02:11 andrei i've got about 5TB of data
02:12 andrei not sure how fast the disks are
02:12 andrei probably about a day
02:12 JoeJulian I think I would mount and find, myself, to ensure they're all synced if I could wait that long.
02:12 JoeJulian Ah, can't wait that long.
02:12 andrei yeah (((
02:12 andrei that's the trouble
02:12 andrei i can't shut things down for one day
02:13 JoeJulian Acquired lock for range 38273024 131072
02:13 andrei by the way, are the changes that i've made to the .vol file going to be permanent?
02:13 JoeJulian As you can see, it's doing the granular locking. That /should/ allow the files to be opened and repair in the background.
02:13 andrei or do I need to remake them at every boot?
02:14 JoeJulian andrei: every boot (right now).
02:15 JoeJulian I'm going to push really hard to get someone to fix this asap. This is seriously broken and kind-of pisses me off tbh.
02:16 andrei so, I guess i should copy these files
02:16 andrei )))
02:16 JoeJulian I want you to file a bug report. Mark it urgent. I'll add myself to the cc list and document everything I can find.
02:16 glusterbot http://goo.gl/UUuCq
02:16 andrei should I file a bug for the port thing that we've discovered?
02:16 glusterbot http://goo.gl/UUuCq
02:17 andrei oh, i see
02:17 andrei okay
02:17 andrei thanks!
02:17 andrei JoeJulian: could you also CC me on it?
02:17 andrei i am keen to get this working asap )))
02:17 JoeJulian That's why you're opening the bug report.
02:17 andrei ah, okay
02:17 andrei will do that
02:18 andrei i will need to create a new login as i've not got one
02:18 JoeJulian I bet m0zes will be ccing himself as well... :)
02:19 JoeJulian I wish we had hardware we could test 3.4 on to see if it has the same problem.
02:20 andrei JoeJulian: could you please give me a hand with filing the bug report?
02:20 andrei i've not done this before )))
02:20 andrei JoeJulian: do you work for redhat doing glusterfs?
02:20 JoeJulian Nope, I work for a small beauty supply distributor in Seattle.
02:21 andrei ah,
02:21 andrei I misunderstood you )))
02:21 JoeJulian So... If you've gone to the file a bug link, you should already have the right product selected.
02:21 andrei my university friend lives in Seattle )))
02:21 glusterbot http://goo.gl/UUuCq
02:21 andrei working for Micro$oft
02:22 JoeJulian component is rdma version is 3.3.0 (since they still haven't added 3.3.1) Summary: rdma does not select the correct port when mounting volume
02:22 andrei i am there
02:22 JoeJulian Then describe it as best you can. Add any logs you still have that you think are relevant.
02:23 JoeJulian Like I say, I'll add my own perspective too, so it should end up pretty clear. Plus raghaven1rabhat knows the rdma code and will probably know before reading it all where the problem probably resides.
02:24 andrei thanks, i will send you the bug number shortly
02:24 JoeJulian glusterbot will announce it too
02:25 andrei while I am on this, shouldn't nfs volumes be automatically exported?
02:25 JoeJulian Yes, but nfs.vol is dealing with the same issue.
02:25 andrei i think in 3.3.0 i had it automatically shown in showmount -e
02:25 andrei ah, so i need to change that as well i guess
02:25 andrei the same applies to replication?
02:26 JoeJulian No, the replica translator is connected to each of the client translators. So once the client translators are connecting the rest works.
02:26 andrei yeah, I can see the second server is being populated with data
02:27 m0zes hooray!
02:27 JoeJulian I would probably try starting up one VM at a time and see what you can get away with.
02:28 JoeJulian fyi, andrei , I am a ,,(volunteer)
02:28 glusterbot A person who voluntarily undertakes or expresses a willingness to undertake a service: as one who renders a service or takes part in a transaction while having no legal concern or interest or receiving valuable consideration.
02:30 JoeJulian If I ever get to GB, you'll owe me a beer.
02:30 andrei JoeJulian: for sure man!
02:30 andrei many beers in fact!
02:31 andrei or if I ever get to Seattle )))
02:31 andrei which I might in the near future to visit my mate
02:31 JoeJulian You could come to the Cascadia IT Conference http://casitconf.org/casitconf13/
02:31 glusterbot Title: Cascadia IT Conference 2013 (at casitconf.org)
02:32 sjoeboo joined #gluster
02:33 JoeJulian When I started into this, I was expecting a simple packaging problem with the rdma library. :D
02:35 JoeJulian Just in time... The boss wants me to go home so he can lock the place up... :D
02:36 sjoeboo joined #gluster
02:38 hagarth joined #gluster
02:39 overclk joined #gluster
02:40 andrei okay, done the bug report
02:41 andrei hehe
02:41 andrei have a good evening
02:44 raven-np joined #gluster
02:46 andrei JoeJulian: what port options should I list in the nfs vol file?
02:46 andrei same as before?
02:50 glusterbot New news from newglusterbugs: [Bug 907695] rdma does not select the correct port when mounting volume <http://goo.gl/W9Pmw>
02:51 andrei does anyone know how I can tell gluster to reread it's nfs .vol file so that it starts the nfs services?
02:56 amccloud joined #gluster
03:03 bharata joined #gluster
03:06 shylesh joined #gluster
03:13 hagarth joined #gluster
03:41 partner joined #gluster
03:52 sjoeboo joined #gluster
04:05 grade_ cant mount gluster vol using this in command in ubuntu: sudo mount.glusterfs gluster:/gv0 /glusterfs
04:05 grade_ err: Mount failed. Please check the log file for more details.
04:16 sripathi joined #gluster
04:22 C2 joined #gluster
04:22 hagarth joined #gluster
04:30 C2 has anyone integrated gluster with Xen as a SDR
04:35 vpshastry joined #gluster
04:39 mkultras is 3.0.0 and 3.0.2 not compatible either?
04:39 mkultras log says either version or transport and transport is tcp on both sides for certain
04:54 lala joined #gluster
05:09 bala joined #gluster
05:10 sgowda joined #gluster
05:10 melanor9 joined #gluster
05:13 raghu joined #gluster
05:13 mkultras yep, they were not compatible it seemed
05:20 srhudli joined #gluster
05:23 bharata joined #gluster
05:23 melanor9 joined #gluster
05:27 killermike joined #gluster
05:28 raghug joined #gluster
05:28 shireesh joined #gluster
05:29 ramkrsna joined #gluster
05:29 ramkrsna joined #gluster
05:36 sripathi joined #gluster
05:39 rastar joined #gluster
05:43 overclk joined #gluster
05:51 bulde joined #gluster
05:57 sripathi joined #gluster
05:59 sripathi1 joined #gluster
06:10 trmpet1 joined #gluster
06:23 bharata joined #gluster
06:29 mohankumar joined #gluster
06:32 rastar joined #gluster
06:48 vimal joined #gluster
07:02 sashko joined #gluster
07:10 trmpet1 joined #gluster
07:14 rastar joined #gluster
07:17 glusterbot New news from resolvedglusterbugs: [Bug 763999] Gluster crashes when starting OpenVZ containers from it <http://goo.gl/Z0Zyg>
07:28 Nevan joined #gluster
07:48 deepakcs joined #gluster
07:48 puebele1 joined #gluster
07:49 rgustafs joined #gluster
07:58 vpshastry joined #gluster
07:59 guigui3 joined #gluster
07:59 ctria joined #gluster
08:02 melanor9 joined #gluster
08:06 puebele joined #gluster
08:09 hybrid5121 joined #gluster
08:19 Humble joined #gluster
08:22 clag_ joined #gluster
08:24 pet joined #gluster
08:25 pet left #gluster
08:27 kevein joined #gluster
08:29 grade___ joined #gluster
08:30 hagarth1 joined #gluster
08:32 vpshastry joined #gluster
08:40 Joda joined #gluster
08:50 dobber joined #gluster
08:54 tjikkun_work joined #gluster
08:59 vpshastry joined #gluster
09:01 Staples84 joined #gluster
09:01 bala joined #gluster
09:03 melanor9 joined #gluster
09:05 venkat_ joined #gluster
09:14 melanor9 joined #gluster
09:22 glusterbot New news from newglusterbugs: [Bug 832632] [Feature Request] Copy On Write (COW) <http://goo.gl/kKjq6>
09:24 dmojoryder joined #gluster
09:25 grade___ hi guys I just cant mount my gluster vol. err is Mount failed. Please check the log file for more details.
09:27 grade___ im using ubuntu as client and centos as server
09:28 grade___ when i mount it from ubuntu client even if I already start fuse. cant still mount the vol
09:31 Staples84 joined #gluster
09:44 stat1x joined #gluster
09:54 haakond joined #gluster
09:55 ninkotech_ joined #gluster
09:58 tryggvil joined #gluster
10:08 hagarth joined #gluster
10:12 al joined #gluster
10:24 bulde joined #gluster
10:25 manik joined #gluster
10:26 vikumar joined #gluster
10:55 shireesh joined #gluster
10:57 mynameisbruce_ joined #gluster
11:02 morse joined #gluster
11:09 ramkrsna joined #gluster
11:13 lh joined #gluster
11:15 vpshastry joined #gluster
11:18 raven-np1 joined #gluster
11:33 duerF joined #gluster
11:36 hagarth joined #gluster
11:50 vpshastry joined #gluster
12:00 mynameisbruce_ can anybody help me on this one
12:00 mynameisbruce_ i try to create an qemu img on gluster
12:00 shireesh joined #gluster
12:00 mynameisbruce_ qemu 1.3.1
12:00 mynameisbruce_ gluster git head
12:01 mynameisbruce_ qemu-img create gluster+tcp://server1-gluster:24007/dis​tributed-replicated-vol/testimage1.img 1G
12:01 mynameisbruce_ also tryed qemu-img create gluster+unix:///distributed-replicated-vo​l/test1.img?socket=/var/run/glusterd.sock 1G
12:01 mynameisbruce_ but everytime i get error "qemu-img: Gluster connection failed"
12:02 mynameisbruce_ i can see socket for tcp connection and i can also see /var/run/glusterd.sock
12:02 mynameisbruce_ any clue?
12:04 mynameisbruce_ both...gluster and qemu is self compiled....qemu with gluster/uuid support
12:14 vimal joined #gluster
12:19 plarsen joined #gluster
12:19 pkoro joined #gluster
12:19 kevein joined #gluster
12:20 mynameisbruce_ okay solved
12:21 mynameisbruce_ volume was called distributed-replicate-vol instead of distributed-replicated-vol  :D
12:21 mynameisbruce_ typo!! i hate that :D
12:23 johndescs :P
12:31 amccloud joined #gluster
12:32 tryggvil joined #gluster
12:32 kkeithley1 joined #gluster
12:36 raven-np joined #gluster
12:36 edward1 joined #gluster
12:51 manik joined #gluster
12:52 andrei joined #gluster
12:55 ramkrsna joined #gluster
12:58 Staples84 joined #gluster
12:59 andrei hello guys
12:59 andrei JoeJulian: thanks a lot for your help yesterday! much appreciated!!!
13:00 andrei guys, could someone please point me in the right direction? I've got an issue with glusterfs 3.3.0 and 3.3.1 as well where the client mounts become frozen while the self healing process takes place?
13:01 andrei is there a way to to fix that?
13:01 andrei i've added a new server to the glusterfs volume and set it to Replicate.
13:01 andrei while the replication takes place the clients are unable to use the filesystem at all
13:06 andrei and i've got 5TB to copy which might take about 25-30 hours
13:06 andrei it's a bit of an issue for me as I can't wait that long while the clients are unable to access their data
13:10 melanor9 hi andrei, i think you might want to read this:  http://joejulian.name/blog/replacin​g-a-glusterfs-server-best-practice/
13:10 glusterbot <http://goo.gl/pwTHN> (at joejulian.name)
13:11 andrei melanor9: thanks i  will take a look at the link
13:11 melanor9 andrei: also if you'd take a look at what  gluster actually does replicating (strace -cfp ) you'd see that its spends most of time in futex(), which is apparently logical bug
13:12 melanor9 you might try to turn  gluster volume set  storage.linux-aio on  to soften the problem
13:14 andrei thanks, I will give it a go
13:14 andrei just reading the blog
13:14 andrei i've spent a bit of time with JoeJulian yesterday trying to resolve the rdma issue with 3.3.1
13:14 andrei we ended up finding a bug and filing it
13:15 andrei so, i have to use 3.3.0 for now
13:18 andrei melanor9: by the way, do you know if I make a small change to a 1TB file would glusterfs replicate only that change across the bricks or will it need to copy the entire 1TB file?
13:18 melanor9 i have to warn about AIO tough, it might be broken  in older linux kernels and in newer kernels ext4 is broken.
13:18 melanor9 what kind of volume you have ?
13:19 melanor9 replica-count  ?
13:19 andrei 2
13:19 andrei i've got 2 servers
13:19 melanor9 it will fully replicate in same subvolume
13:19 andrei Number of Bricks: 1 x 2 = 2
13:20 andrei Type: Replicate
13:20 melanor9 i had an experiment with gluster nearby, copying brick with tar and feeding it to gluster
13:21 melanor9 and if there was not much of an access, gluster accepted the new brick without hitch
13:21 melanor9 but i'm not sure if its a valid tactic
13:22 melanor91 joined #gluster
13:25 kamus joined #gluster
13:25 melanor92 joined #gluster
13:26 kamus morning
13:26 andrei joined #gluster
13:26 kamus it's ok if I ask for some help here?
13:27 andrei melanor: what do you mean by fully replicate? does that mean that each time a 1TB file changes a bit it will copy the entire file across to the other server?
13:27 andrei kamus: go ahead, that's what this chanel is all about
13:28 andrei people are friendly and will try to help if they know how
13:28 andrei or have time
13:28 andrei or both ;-)
13:28 kamus heh, I had a 2 replica setup yesterday that was working fine until I added two new bricks
13:29 kamus right now I have a distribuited replicated volumen with 4 bricks (2 bricks from server A and 2 from B)
13:29 melanor92 it will not copy the whole file ofc, but  what you do ?  append/insert ?
13:29 andrei okay
13:29 andrei melanor92: well, the large files are qcow2 vm images. s
13:30 andrei not sure if it appends or changes contents
13:31 kamus the problem is if I try to mount the volumen (for instance volume1) I cannot my files
13:31 andrei i would imagine once the vm volume reaches it's max size it will simply replace some contents instead of appending
13:31 melanor92 then i guess it will be copied over
13:31 andrei kamus: do you see a blank file system?
13:31 kamus something happened after add-brick and rebalance commands were executed
13:32 kamus I see some part of the structure, but not all the files (the most important ones by the way)
13:33 kamus in addition, I can see a correct size of my new volumen, but not all the files
13:33 bala1 joined #gluster
13:33 kamus a important note is that I have added this two new bricks from an older folder (before add to new volumen I deleted all the data, not a format)
13:33 andrei melanor92: by the way, I can't seems to find storage.linux-aio option when i do gluster volume set help
13:34 melanor92 its interesting is glusterbot knows about  afr-self-heal-data.c:712:afr_sh_data_fix
13:34 andrei is that an undocumented feature
13:34 kamus seems like this two new bricks have not been replicated correctly or something
13:35 andrei kamus: did the replication process finish already?
13:35 melanor92 nada
13:35 andrei perhaps it's still doing it and that could be the reason why you are not seeing full structure?
13:35 melanor92 andrei: its rather new :)
13:35 kamus andrei, I have not forced a sync for the volume.. I have just added the bricks and run rebalance commands
13:36 melanor92 http://www.gluster.org/community/doc​umentation/index.php/GlusterFS_3.3.1
13:36 glusterbot <http://goo.gl/f03sM> (at www.gluster.org)
13:36 andrei kamus: there is a command gluster volume heal
13:36 andrei you might want to run it
13:36 kamus I have not forced a sync because I don't want to loose that files
13:36 melanor92 storage/posix: implement native linux AIO support
13:36 melanor92 Configurable via cli with "storage.linux-aio" settable option
13:36 melanor92 Backported Avati's patch http://review.gluster.org/#change,3627
13:36 glusterbot Title: Gerrit Code Review (at review.gluster.org)
13:37 kamus ho, I forgot to mention that I am running 3.3.1-1
13:37 andrei melanor92: ah, I am afraid i can't try that one becuase of the rdma bug in 3.3.1
13:37 kamus in a Debian box
13:37 andrei which makes it unusable for me
13:37 kamus andrei, is it dangerous to run a heal? I mean, exists any possibility to lost some data?
13:39 andrei kamus: I am not the right person to ask that unfortunately. I am the beginner as well
13:40 andrei kamus: you can try running the command in info mode
13:40 kamus andrei, don't worry :) thanks any way
13:40 andrei which should tell you what the differences are
13:40 andrei do gluster volume heal help
13:41 kamus certainly there is a diference in a brick
13:41 kamus for a directory
13:41 andrei the heal command should copy the files across the servers
13:41 kamus that exactly contains all the files that I need to be visible
13:41 andrei and your fs should be intact
13:42 andrei perhaps someone else from here can confirm that
13:42 andrei I am currently trying to heal 2 servers
13:42 andrei with about 5tb of data
13:42 kamus I will check that
13:42 andrei and I am having issues with client mountpoints becoming frozen
13:42 andrei while the healing takes place
13:42 kamus uhm..
13:43 andrei so, you might watch out for that
13:43 kamus this is a production system so..
13:43 andrei same here mate
13:43 kamus I will look for more information :)
13:43 andrei i've got a production system as well
13:43 andrei so i was doing it from about 11pm to 4am )))
13:43 kamus :o
13:43 andrei there are sacrifices one has to make ;-)
13:45 andrei kamus: if that happens to you, I would suggest that you temporarily disable the new bricks by switching off the glusterfs
13:46 andrei i found that as soon as i kill all glusterfs processes on the second server (the one that receives the changes)
13:46 andrei once this is done, the mountpoints become active once again
13:46 kamus have you ever remove 2 bricks from a 4 replica setup?
13:46 andrei nope, i've got 2 servers only
13:47 dustint joined #gluster
13:47 andrei the old server which is a production env
13:47 andrei and the new server we've got for data replication
13:47 kamus seems like the only way to rollback to my old setup and check what's happening
13:47 andrei this is what I am currently doing
13:48 andrei i guess I need about a week of sleepless nights to copy 5TB of data
13:48 melanor92 can you turn your original bricks to RO ?
13:48 andrei unless someon could point me in the right direction how to make the filesystem not freeze
13:49 andrei while the replication takes
13:49 andrei place
13:52 kamus I tried to run a heal on the volumen and in glusterd.log there is a "Skipping entry self-heal because of gfid absence"
13:55 kamus I don't have a problem to remove those new 2 bricks, while my data continues in old bricks I am fine
13:56 chirino left #gluster
13:59 mohankumar joined #gluster
14:01 aliguori joined #gluster
14:04 hybrid5121 joined #gluster
14:21 rwheeler joined #gluster
14:22 balunasj joined #gluster
14:29 manik joined #gluster
14:35 hybrid5121 joined #gluster
14:47 Staples84 joined #gluster
14:52 stopbit joined #gluster
14:52 jskinner_ joined #gluster
14:53 jskinner_ I'm having some issues trying to get a Gluster exported volume to be used with KVM
14:53 jskinner_ anyone have some ideas on this?
14:55 jskinner_ http://pastebin.com/ffppwhXv
14:55 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
15:01 noob2 joined #gluster
15:02 kamus joined #gluster
15:12 andrei jskinner: hi
15:13 andrei i've been doing a bit of testing with kvm and gluster a while back
15:14 andrei jskinner_: i think your issue relates to how kvm tries to create the file
15:14 andrei it probably uses one of the flags which are not working well with gluster
15:14 melanor9 joined #gluster
15:14 andrei there is an option to use use different caching method while creating the volume
15:15 jskinner_ yeah I've read something on that.
15:15 andrei i've fixed that issue in the past
15:15 jskinner_ I'm wondering if it's a permission issue
15:15 andrei there is even a library out there that allows you to substitute one call for another
15:15 andrei jskinner_: nope, that shouldn't be a permission issue
15:15 x4rlos Anyone done a good writeup with KVM and gluster? I am very interested in this option. Especially if LVM is thrown into the mix.
15:16 andrei however, I can kill your hopes right now
15:16 andrei you will see very very poor performance for now
15:16 jskinner_ lol, and I was off to such a good start to my day
15:16 andrei at the moment kvm + glusterfs doesn't work together well
15:16 andrei however
15:17 andrei they are releasing version 3.4 which should address the issue
15:17 wushudoin joined #gluster
15:17 andrei together with the latest kvm it will work very nicely
15:17 jskinner_ Yeah I've seen that. Any eta on that release at all?
15:17 andrei no idea, I am also interested in that
15:17 jskinner_ haven't seen it anywhere on the website
15:17 andrei in my example, the pure glusterfs mountpoint performance was around 700-800MB/s
15:17 jskinner_ hmmm
15:18 andrei the vm running on that fs was only giving me 50mb/s at best
15:18 andrei )))
15:18 jskinner_ dang
15:19 andrei http://www.gluster.org/2012/​11/integration-with-kvmqemu/
15:19 glusterbot <http://goo.gl/IhqoH> (at www.gluster.org)
15:19 andrei this should point you in the rigth direction
15:19 andrei http://raobharata.wordpress.com/2012/10​/29/qemu-glusterfs-native-integration/
15:19 glusterbot <http://goo.gl/f2MhH> (at raobharata.wordpress.com)
15:20 andrei x4rlos: sorry i've given up on kvm + gluster for now
15:20 andrei jskinner: what you can do is to use glusterfs over nfs for the time being
15:20 andrei that will give you a decent level of performance
15:20 jskinner_ ok that was my next question
15:20 x4rlos hehe. Okay. 3.4 is on debian experimental repo, if it's supposed to be much better performance.
15:20 andrei and switch to native gluster when 3.4 is out and becomes stable
15:21 jskinner_ what kind of performance did you see with NFS as opposed to native?
15:21 andrei x4rlos: could you give me a link to that? I am using ubuntu, do you know if there is a ppa for that?
15:21 andrei native glusterfs was giving me around 700-800mb/s mark when using iozone
15:22 andrei gluster over nfs is giving me around 400-500 mb/s mark
15:22 andrei however, this is done using rdma as the transport
15:22 andrei not tcp
15:22 jskinner_ hmm
15:22 andrei nfs is served over ipoib
15:23 jskinner_ are there any major limitations in using NFS over native?
15:23 andrei so it could be the difference between IP and RDMA that's causing the difference and not the gluster native vs nfs
15:23 andrei yes, there is
15:23 bugs_ joined #gluster
15:23 andrei for once, nfs is not aware of all of your servers
15:24 andrei so the client connects just to a single glusterfs server
15:24 jskinner_ ahh ok
15:24 andrei whereas glusterfs is aware of all of your servers
15:24 jskinner_ well crap
15:24 jskinner_ that is a huge limitation lol
15:24 andrei jskinner: there is an nfs HA setup that you can use )))
15:24 andrei using virtual IPs
15:25 andrei like UCARP
15:25 andrei something like that: http://www.debian-administration.o​rg/article/678/Virtual_IP_addresse​s_with_ucarp_for_high-availability
15:25 glusterbot <http://goo.gl/fumTJ> (at www.debian-administration.org)
15:25 andrei or other methods
15:25 jskinner_ yeah I've looked at ucarp in the past
15:25 andrei that way if one server goes down the clients should still work
15:25 andrei jskinner_: how did it go?
15:25 andrei iv'e had a quick look at that a few months back and it didn't work for some reason
15:26 andrei as it wasn't a priority, i've given up
15:26 andrei however, I still need to come back to it
15:26 andrei as I am now introducing a second server to my glusterfs setup
15:26 _br_ joined #gluster
15:26 jskinner_ It worked ok - but when I was doing failure testing. Most of my VMs would drop into a read-only fs
15:27 jskinner_ when I would kill a connection to one of the storage servers
15:27 andrei ah,
15:27 andrei very interested to know more
15:27 andrei did you create a howto while you were doing that?
15:27 jskinner_ rebooted the VMs, and they would come back fine
15:27 jskinner_ but unacceptable in my opinion
15:27 andrei yeah, I want to avoid this as well
15:27 jskinner_ I did not
15:27 jskinner_ was just toying with it
15:28 andrei have you tried changing the failover times?
15:28 jskinner_ and that was in a smaller lab then what I am trying now
15:28 andrei perhaps you've waited for too long?
15:28 jskinner_ could be
15:28 jskinner_ this was 6+ months ago
15:28 jskinner_ and now we are looking at gluster again
15:28 jskinner_ but trying to stick with native
15:28 andrei jskinner: are you using openstack?
15:28 jskinner_ cloudstack
15:28 jskinner_ apache
15:28 andrei )))
15:28 Humble joined #gluster
15:28 andrei nice one
15:29 andrei same as me mate!
15:29 andrei where are you based?
15:29 jskinner_ I prefer it to openstack
15:29 jskinner_ Midwest
15:29 andrei US?
15:29 jskinner_ yeah
15:29 andrei that would be a long way to travel for a meet ))
15:29 jskinner_ was just at the cloudstack colab' back in December
15:30 jskinner_ lol
15:30 andrei I am in London
15:30 jskinner_ nice - never been
15:30 amccloud joined #gluster
15:31 _br_ joined #gluster
15:31 andrei anyway
15:32 andrei what i've found is you will need to recompile cloudstack in order to change how kvm creates files
15:32 andrei so, even if glusterfs + kvm would play nicely
15:33 andrei you will still need to cloudstack to know about it and use the extra options which are required to utilise direct kvm/gluster integration
15:33 hchiramm_ joined #gluster
15:33 andrei and that is I am not sure how to do at the moment
15:33 andrei i've asked the #cloudstack but not received any comments
15:34 jskinner_ hmm
15:34 jskinner_ I was just planning on using shared mount points from cloudstack side
15:34 andrei jskinner: i don't think how the native gluster+kvm will work
15:34 andrei you need to read the blog
15:34 jskinner_ hmm
15:35 andrei they give the examples how you need to create images in order to use the direct method
15:35 jskinner_ interesting
15:35 andrei thus, the shared mountpoint will not be enough
15:36 andrei however, cloudstack is opensource, and perhaps it's not that difficulat to add additional lines and recompile it
15:36 jskinner_ Well that rains on my parade. I figured since I was just mounting the export in the same location on each kvm host it would just work from cloudstack side lol
15:36 jskinner_ as that's how I deploy GFS2
15:36 jskinner_ but I suppose GFS2 and Gluster are just a bit different lol.
15:37 andrei how are you finding gfs2?
15:37 andrei i've not yet tested it?
15:37 andrei what is your experience?
15:38 andrei one of the reasons why i am going for gluster is the rdma support. we use infiniband
15:38 aliguori joined #gluster
15:39 jskinner_ Just started doing GFS2 testing, so I haven't had a chance to load it up yet.
15:39 jskinner_ haven't had any issues yet
15:39 Staples84 joined #gluster
15:39 jskinner_ vm creation is snappy. Migration to different nodes works good, snapshotting, etc.
15:40 jskinner_ The GFS2 is on an iSCSI 10gb backend
15:40 jskinner_ basically doing the gfs2 and gluster testing side by side
15:45 sjoeboo joined #gluster
15:46 andrei joined #gluster
15:46 andrei sorry got disconnected
15:50 jskinner_ it happens :)
15:50 hchiramm_ joined #gluster
15:52 andrei so, what was your experience with gfs2
15:54 jskinner_ Haven't had any issues
15:54 jskinner_ things seemed to be pretty snappy
15:54 jskinner_ The GFS2 is on an iSCSI 10gb backend
15:54 jskinner_ haven't done any load testing on it yet though
15:54 jskinner_ so that will be interesting
15:55 jskinner_ only basic stuff so far i.e., vm creation, snapshotting, templating, vm migration.
16:00 andrei jskinner_ are you using kvm?
16:01 andrei how does snapshotting work for you?
16:01 daMaestro joined #gluster
16:01 balunasj joined #gluster
16:03 jskinner_ yeah that would be with kvm
16:03 jskinner_ snapshotting was working fine
16:04 jskinner_ it was quick
16:04 Dave2 joined #gluster
16:05 jskinner_ for the snapshots, i was just running a Nexenta box with the drives configured in a raidz2 setup and exported the share as nfs to cloudstack
16:06 jskinner_ kvm seemed to handle the snapshotting well
16:06 andrei i had an issue with snapshotting speed
16:07 andrei as I am using infiniband for storage and gigabit for public/guest/management
16:07 jskinner_ hmm
16:07 andrei in my case I can't make the secondary storage server to run over infiniband
16:07 jskinner_ ah
16:07 andrei the way that the cloudstack designed the secondary storage concept
16:07 jskinner_ it was pretty snappy for me, however, the environment was idle.
16:11 nueces joined #gluster
16:18 sjoeboo joined #gluster
16:20 raven-np1 joined #gluster
16:22 andrei okay, cool
16:22 andrei it would be very interesting to check how gluster works over 10G network
16:22 andrei what sort of throughputs you should expect
16:22 manik joined #gluster
16:22 andrei jskinner: by the way, what hardware do you have on your storage servers?
16:31 jskinner_ for hardware, it's commodity. they are supermicro chassis and boards, with 144GB of RAM 2 Intel Xeon quad core CPUs, and 14 1TB SAS drives.
16:31 jskinner_ 4 10GB interfaces
16:31 jskinner_ 2 for LAN, and 2 for the SAN
16:31 jskinner_ utilize bonding mode 5
16:32 jskinner_ three of those boxes to create the Gluster cluster for testing.
16:33 elyograg For my testbed, I bought six beige (black actually) boxes for $300 each, we reused old 1TB SATA disks, four per server.
16:33 elyograg newegg component special. ;)
16:33 jskinner_ nice
16:34 andrei very nice indeed
16:34 andrei )))
16:34 andrei are you running zfs on them?
16:34 jskinner_ I am not running ZFS with Gluster
16:34 jskinner_ I am on my GFS2 environment
16:35 jskinner_ this is xfs
16:35 jskinner_ for now
16:35 elyograg xfs.  centos.  i'm planning to use CentOS for the storage nodes and Fedora 18 for the network access nodes.
16:35 andrei i am using pretty similar hardware, but 24gb of ram and some ssds for log and caching
16:35 jskinner_ brtfs one day, to replace xfs
16:35 andrei btrfs is very nice indeed
16:36 andrei actually, you may also take a look at ceph
16:36 jskinner_ I am running CentOS 6.3 across the board
16:36 andrei as i've tested it as well and it was pretty okay
16:36 jskinner_ I have looked at ceph
16:36 Staples84 joined #gluster
16:36 andrei i've not done any exteded tests
16:36 Humble joined #gluster
16:36 andrei as it doesn't work over rdma
16:36 jskinner_ I've heard it doesn't scale as well as they claim
16:36 jskinner_ but would like to see for my self lol
16:37 jskinner_ the thing about ceph that puts me off is all of the components
16:37 jskinner_ it seems over-engineered
16:37 jskinner_ but I am no storage expert, so what do I know lol.
16:38 andrei )))
16:38 andrei i've not gone through much details and just followed a quick howto
16:39 andrei it worked like a charm and got installed and working on 4 vm machines in a matter of minute or so
16:39 jskinner_ I may look at it again
16:39 jskinner_ probably not though to be honest lol
16:40 jskinner_ im holding out for gluster
16:40 andrei yeah, same as me
16:40 _br_ joined #gluster
16:40 andrei gluster should be very nice once you get a lot support from RH
16:41 jskinner_ yeah
16:41 jskinner_ CentOS is my preferred distro so it works out well for me lol
16:42 andrei okay
16:42 andrei i am more of an ubuntu guy and
16:42 andrei have to use centos because of the crappy driver support for infiniband kit that we've got
16:42 jskinner_ lol
16:42 jskinner_ ive never worked with infiniband
16:42 andrei so, i've got servers running ubuntu and clients running centos
16:43 andrei oh, it's a beauty and beast in one )))
16:43 andrei if you are going to, never purchase qlogic ib cards
16:43 andrei never
16:43 andrei it's such a pain in the bum
16:43 andrei unreal
16:44 andrei go for Mellanox kit
16:44 jskinner_ will keep that in mind
16:46 andrei by the way, does anyone know if I can simply copy the contents of one server to another instead of doing the self healing?
16:46 _br_ joined #gluster
16:47 andrei i've got a trouble with self healing that it simply freezes mountpoints on the clients
16:47 andrei and I can't afford any downtime
16:48 andrei so i thought of copying files across manually
16:48 andrei would this work?
17:00 hagarth joined #gluster
17:05 Humble joined #gluster
17:08 melanor9 joined #gluster
17:09 theron joined #gluster
17:09 erik49 can yone hep me debug why i can't mount a volume?
17:09 erik49 http://dpaste.com/910003/ <-- error log
17:09 glusterbot Title: dpaste: #910003 (at dpaste.com)
17:10 erik49 i have another instance that has mounted the volume
17:10 erik49 so am pretty confused
17:15 cicero firewall issue by chance?
17:16 ndevos erik49: maybe you can mount with more verbose logging? like: mount -t glusterfs -o log-level=TRACE server:/vol /mnt
17:17 jbrooks joined #gluster
17:17 balunasj|mtg joined #gluster
17:19 erik49 ndevos, http://dpaste.com/910007/
17:19 glusterbot Title: dpaste: #910007 (at dpaste.com)
17:21 ndevos erik49: like cicero I guess it's firewall or network related... got something in the glusterd log?
17:22 ndevos erik49: there is no real timeout, so the connection is probably terminated on purpose
17:23 ndevos erik49: do the versions of your client and server match?
17:23 erik49 nothing that looks useful
17:24 erik49 how do i check glsuter client version?
17:26 ndevos erik49: just the version of the packages you have installed 3.3.1 is what you have on the client, you would need to use that on the server too
17:31 erik49 ndevos, thats it thanks!
17:32 _br_ joined #gluster
17:39 _br_ joined #gluster
17:41 _br_ joined #gluster
17:43 hchiramm_ joined #gluster
17:43 _br_ joined #gluster
17:55 balunasj joined #gluster
18:06 Humble joined #gluster
18:06 _br_ joined #gluster
18:10 isomorphic joined #gluster
18:13 _br_ joined #gluster
18:13 the-me kkeithley: ping
18:13 _br_ joined #gluster
18:13 jbrooks joined #gluster
18:14 Humble joined #gluster
18:23 pandagerbil left #gluster
18:27 ctria joined #gluster
18:28 sjoeboo joined #gluster
18:30 sashko joined #gluster
18:32 jbrooks joined #gluster
18:36 erik49 ugh
18:36 erik49 i just upgraded to gluster 3.3
18:36 erik49 and am having trouble recreating my volume
18:36 _br_ joined #gluster
18:37 erik49 glusterd –xlator-option *.upgrade=on -N does not work
18:38 JoeJulian Define "not work"
18:38 erik49 gives my usage error
18:38 erik49 me*
18:39 JoeJulian Oh, I see... you copied and pasted from a wordpress page.
18:39 JoeJulian Check you hyphens.
18:39 JoeJulian your
18:39 erik49 ahhh
18:39 JoeJulian If you upgrade using the yum repo, btw, that happens for you.
18:39 erik49 i used apt-get
18:40 erik49 lol that would have taken me foreevver to debug
18:40 erik49 hmm
18:40 erik49 still get the error
18:41 erik49 bricks "or a prefix of it is already part of a volume"
18:41 glusterbot erik49: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
18:41 erik49 am i going to be able to recover my files if i do the setfattr thing?
18:41 JoeJulian yep
18:43 _br_ joined #gluster
18:44 hateya joined #gluster
18:45 jiqiren joined #gluster
18:45 erik49 thanks!
18:46 JoeJulian You're welcome.
18:53 juhaj m0zes: You mentioned multi-master georeplication some time ago. I cannot find anything regarding that on the web pages. What is it exactly?
18:53 Humble joined #gluster
18:58 nueces joined #gluster
18:58 erik49 still getting /exports/brick17 or a prefix of it is already part of a volume
18:58 glusterbot erik49: To clear that error, follow the instructions at http://goo.gl/YUzrh or see this bug http://goo.gl/YZi8Y
18:59 erik49 did all the setfattr stuff \=
18:59 erik49 getfattr -d returns nothing
19:00 erik49 and ive restarted glusterd
19:00 _br_ joined #gluster
19:02 erik49 there are no .glusterfs dirs either
19:02 erik49 http://dpaste.com/910075/
19:02 glusterbot Title: dpaste: #910075 (at dpaste.com)
19:03 _br_ joined #gluster
19:04 balunasj joined #gluster
19:08 m0zes juhaj: multi-master georeplication is a feature that *was* slated for 3.4. it doesn't look like it is going to make it in time.
19:09 m0zes with the current geo-replication setup, the sync is only one direction. with multi-master, the sync could be n-direction, you could have n sites with a full *writeable* copy in each site.
19:12 _br_ joined #gluster
19:16 erik49 help? i can't recover my filesystem :(
19:17 andrei joined #gluster
19:18 m0zes find ./ -name '.glusterfs'
19:24 juhaj m0zes: That would be very nice indeed!
19:28 jbrooks joined #gluster
19:37 JoeJulian erik49: getfattr -d isn't going to show the trusted.* attributes. See what you get if you 'getfattr -m . -d -e hex'
19:39 andrei JoeJulian: hello ))
19:39 andrei nice to see you )))
19:40 JoeJulian Hey, there he is... :)
19:42 JoeJulian Ok, so you did roll back to 3.3.0 then?
19:43 andrei joined #gluster
19:43 andrei sorry to have kept you for ages yesterday
19:44 erik49 hmm i guess i had to umount the volume first \=
19:44 andrei without your help I wasn't able to figure out how to restart the nfs service so that gluster would start serving nfs shares
19:44 andrei after we've fixed the port issue
19:44 JoeJulian andrei: Glad to do it. I hate bugs like that.
19:44 andrei yeah
19:44 JoeJulian Actually, I did check in later, but you were offline. :/
19:45 andrei aparently there is a bug report already about the ports
19:45 andrei it's been submitted in 2011 i think
19:45 andrei relating to the same thing
19:45 andrei however, it must have been fixed for some time
19:45 andrei as 3.3.0 is okay
19:45 andrei but 3.3.1 is buggered again ;-)
19:46 JoeJulian Well it seems that most of the team is in Bangalore this week so that could be good.
19:47 andrei )))
19:47 andrei anyways, yesterday i had to revert back to 3.3.0 as i had to bring infrastructure back online
19:48 andrei and an hour ago I had a kernel panic caused by glusterfsd process (((
19:52 JoeJulian Kernel panic??? I blame the kernel devs then.
19:53 JoeJulian That's weird, though. A userspace process should not be able to cause a kernel panic.
19:54 Humble joined #gluster
20:02 zaitcev joined #gluster
20:02 rpetre joined #gluster
20:05 rpetre hello, a quick question (probably a faq, but can't find a definite answer anywhere): is a separate lan for inter-server communication supported? meaning, use different ips (and interfaces) for replication and s2s communication than c2s
20:05 amccloud joined #gluster
20:06 semiosis rpetre: there's only s2s communication when you use nfs clients
20:06 semiosis at least for the bulk of the data xfer.  there's some cluster mgmt traffic but that's tiny
20:07 rpetre semiosis: oh, makes sense
20:07 semiosis right now the best way to achieve it is using split-horizon dns (or worse, /etc/hosts files) to map the same server hostname to a different IP depending on whether it's being resolved from a server or a client
20:07 plarsen joined #gluster
20:08 rpetre let's say that my religion looks down upon hosts files :)
20:10 andrei JoeJulian: yeaht, that's right
20:10 andrei it shouldn't really happen
20:10 andrei )))
20:10 rpetre well, right now i'm using a single (SPOF-y) nfs server with a few clients, i'm looking to replace it with a pair of gluster machines with crosslink to save lan traffic, if the gluster nfsv3 implementation isn't particularly crappy compared to the kernel one, it would work
20:11 semiosis rpetre: sounds like a good plan, although you ought to consider switching to glusterfs native fuse clients too, for your linux clients
20:12 semiosis rpetre: you get automatic HA with that
20:12 semiosis otherwise nfs ,,(mount server) is still a spof
20:12 glusterbot (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
20:12 semiosis unless you go to the trouble of setting up a VIP
20:12 rpetre semiosis: that's what i was planning, but i'm afraid of hitting a bandwidth cap in the lan
20:13 semiosis ah ok
20:13 rpetre so i planned to throw a crosslink between the servers
20:13 semiosis rpetre: is your workload very read-heavy or more balanced?
20:13 rpetre tbh, i'm not entirely sure
20:13 semiosis ok
20:14 rpetre it's the typical "let's add more webservers but make them have the same files" scaling story :)
20:15 rpetre i started to split stuff based on role, but i haven't done yet any stats
20:15 semiosis just in case... ,,(php)
20:15 glusterbot php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
20:16 rpetre i suspect read mostly, but occasionally there are some temp files and whatnot
20:16 rpetre sounds like my case, thanks
20:16 semiosis yw
20:16 semiosis right off the bat nfs will probably perform better than fuse clients due to caching in the kernel nfs client
20:16 rpetre i can probably push for distributed code release on local disks
20:18 rpetre we use phpexpress, i think it's related to apc, i'll pass the problem to the dev team
20:18 rpetre (maybe it'll finally convince them that winscp-based releases are not a great idea)
20:21 jbrooks joined #gluster
20:22 rpetre anyway, thanks, i'll probably drop by for more questions in the coming weeks as i make the test env :)
20:23 rpetre good night/afternoon/morning accordingly :)
20:23 rpetre left #gluster
20:25 jskinner joined #gluster
20:38 szopa joined #gluster
20:46 noob2 left #gluster
20:46 jbrooks joined #gluster
20:52 sashko joined #gluster
20:53 amccloud joined #gluster
20:55 jack joined #gluster
20:55 jskinner_ joined #gluster
20:59 erik49 is there a reason why the apt-repository still installs gluster 3.2?
21:04 _benoit_ Is there a way to do server failover when using the qemu gluster code ?
21:04 semiosis erik49: what apt-repository?  ,,(ppa)
21:04 glusterbot erik49: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
21:21 bronaugh hey; how does one remove a brick from glusterfs entirely w/o taking down each glusterd in the cluster?
21:32 amccloud joined #gluster
21:37 polenta joined #gluster
21:45 cyberbootje joined #gluster
21:48 lanning joined #gluster
21:54 amccloud joined #gluster
22:02 amccloud joined #gluster
22:06 erik49 the precise
22:06 erik49 repository
22:15 hattenator joined #gluster
22:16 y4m4 joined #gluster
22:23 dmojoryder if I shutdown glusterd on a server that participates in a 2 node replicated volume, I assume that when restart glusterd it automatically re-participates in volumes it was already configured for, is that correct? But if I shutdown glusterd on both nodes, when restarted I have to redo the peer probe and the gluster volume create and start?
22:24 phox joined #gluster
22:24 semiosis erik49: there are reasons yes, just use the ,,(ppa) and apt-get on with your life :)
22:24 glusterbot erik49: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
22:25 erik49 yeah i did :D
22:25 erik49 thanks though
22:25 elyograg dmojoryder: that should not be necessary if you haven't changed anything.
22:25 semiosis dmojoryder: you should try it
22:25 semiosis and see
22:25 semiosis but yeah elyograg is right, it should "just work"
22:27 phox so it appears that I'm seeing performance issues accessing files that aren't already in cache, reading at about 150MB/sec over a single 5GB file vs 600MB/sec off the parent FS where the brick actually lives.  thoughts on hopefully doing something about this?
22:28 phox FWIW this is local, not remote
22:28 Ryan_Lane joined #gluster
22:31 semiosis phox: ,,(pasteinfo)
22:31 glusterbot phox: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
22:32 phox http://dpaste.org/gVNQ6/
22:32 glusterbot Title: dpaste.de: Snippet #218552 (at dpaste.org)
22:33 phox tada.
22:33 phox also, obviously it can't be bothered saying there, but it's living on top of ZFS
22:34 semiosis phox: comparing a single brick glusterfs volume to a locally mounted filesystem is not really useful
22:34 raven-np joined #gluster
22:34 phox semiosis: I don't see why
22:34 semiosis if you dont need a distributed cluster filesystem, don't use one
22:35 phox semiosis: then what do you propose I/we use :P
22:35 semiosis nfs?
22:35 phox semiosis: NFS is broken.  this provides similar functionality.
22:35 phox clearly you haven't used NFS enough.  as old as it may be, it's still the buggiest piece of crap ever =/
22:36 phox at least at this point this is exceeding NFS/ethernet performance and is probably stable... but I'm surprised it's choking so badly on a single file like that...
22:38 semiosis phox: well glusterfs targets distributed use cases, with 2+ servers
22:38 semiosis replication between servers, distribution over servers, like that
22:39 phox semiosis: still one would hope that it would attain reasonable performance given the combined performance of the underlying filesystems
22:39 semiosis it really shines when you use it to make a storage cluster serving lots of files to lots of client machines
22:39 phox semiosis: and I do doubt it scales better than linearly...
22:39 phox yeah.  good parallelism.  I'm just wondering if there's some tuneable read-ahead or something smart I can make it do to not fail so badly on large files, which is usually a trivial case for most filesystems.
22:41 semiosis you can see tunable options with 'gluster volume set help' iirc
22:41 phox k
22:41 semiosis but tbh i wouldn't get my hopes up
22:41 semiosis if you do have success though please let us know
22:44 * phox waits for test
22:44 phox that sucks less.
22:44 phox root@skateboard:~# gluster volume set datasets-0099 performance.read-ahead-page-count 16
22:45 semiosis less suck = more win?
22:45 semiosis progress
22:45 phox so I guess the question is why is the maximum value for that parameter so goddamn low?
22:45 phox 16 pages?
22:45 phox seriously, only 64k readahead on 4T disks?  what the smeg? :)
22:45 phox I was going to opt for 64 pages, but it wouldn't allow that
22:45 phox thanks for pointing me in the right direction there
22:46 phox brb.
22:46 semiosis yw
22:49 amccloud joined #gluster
23:03 bronaugh so yeah. MOAR PAGE READAHEAD
23:04 semiosis feel free to file a bug with your request
23:04 glusterbot http://goo.gl/UUuCq
23:07 phox "Gere's a patch.  I've implemented it as a regular expression, to be run against the source." :P
23:12 semiosis ,,(hack)
23:12 glusterbot The Development Work Flow is at http://goo.gl/ynw7f
23:25 amccloud joined #gluster
23:28 jabrcx joined #gluster
23:31 jabrcx hello.  Should I expect flock(2) to work on an nfs client mounting gluster 3.3.1, using the gluster nfs server?
23:37 jabrcx nvm, I believe I have answered my own question, the answer is "yes"... it's working on some other clients
23:48 nueces joined #gluster
23:58 andrei anyone alive? was wondering if there is a way to add async,no_root_squash options to the nfs server?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary