Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-25

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:14 JoeJulian shoot away
00:14 JoeJulian slabgrha: ^
00:20 angband joined #gluster
00:20 angband derp
00:20 angband JoeJulian - i got disconnected... knew that would happen
00:20 angband brb
00:24 fleducquede joined #gluster
00:25 jbrooks joined #gluster
00:49 jbrooks joined #gluster
00:58 slabgrha joined #gluster
00:58 slabgrha joined #gluster
00:59 slabgrha we have an app that uploads logos - which currently gets rsynced to a bunch of front-end web servers.   i'd like to provide some shared storage for that.  Is gluster suited for that workload?
00:59 slabgrha i.e. the app server(s) have a gluster volume mounted and presented to the front ends as well?
01:01 JoeJulian What's the latency like between the storage and the app servers?
01:02 yinyin joined #gluster
01:02 yinyin joined #gluster
01:02 yinyin joined #gluster
01:06 fleducquede joined #gluster
01:58 social joined #gluster
02:19 bala joined #gluster
02:49 bharata joined #gluster
03:01 bdperkin joined #gluster
03:36 zwu joined #gluster
03:52 itisravi joined #gluster
03:54 vpshastry joined #gluster
04:00 _pol joined #gluster
04:18 hflai joined #gluster
04:20 vpshastry joined #gluster
04:30 mohankumar joined #gluster
04:33 hagarth joined #gluster
04:36 y4m4 joined #gluster
04:40 aravindavk joined #gluster
04:43 angband joined #gluster
04:44 shylesh joined #gluster
04:46 bala1 joined #gluster
04:58 mohankumar joined #gluster
05:10 shylesh joined #gluster
05:14 bulde joined #gluster
05:17 shireesh joined #gluster
05:19 red_solar joined #gluster
05:20 bulde1 joined #gluster
05:22 sgowda joined #gluster
05:28 saurabh joined #gluster
05:31 rastar joined #gluster
05:41 satheesh joined #gluster
05:50 satheesh joined #gluster
05:51 glusterbot New news from newglusterbugs: [Bug 952029] Allow an auxiliary mount which lets users access files using only gfids <http://goo.gl/x5z1R>
05:53 phase5 joined #gluster
05:54 phase5 left #gluster
05:55 lalatenduM joined #gluster
06:14 lalatenduM joined #gluster
06:21 glusterbot New news from newglusterbugs: [Bug 949406] Rebalance fails on all the nodes when glusterd is down on one of the nodes in the cluster <http://goo.gl/Q8dyW>
06:22 bulde joined #gluster
06:22 hagarth joined #gluster
06:28 rastar joined #gluster
06:31 ricky-ticky joined #gluster
06:34 guigui1 joined #gluster
06:35 ollivera joined #gluster
06:36 vimal joined #gluster
06:36 vshankar joined #gluster
06:46 hybrid5122 joined #gluster
06:52 sgowda joined #gluster
06:58 puebele1 joined #gluster
07:06 koubas hi, i have 3.3.1, 1 x 2 replicated setup with 200GB, 100 000s small files (web images)  and 100s files 10MB - 2GB, mainly logs, which are regularly appended with small amounts of data. I'm struggling with "broken" replica, that is now turned off, for about a week. My main problem is, that whenever i turn the replica on, the "OK" brick's cpu (4 cores) and IO (raid1, 2x7200 disks ) get absolutely saturated and fuse mounts on two clients almost inacc
07:06 koubas essible . Tonight I updated clients and servers to 3.3.2qa2, without any improvement. I also tried to move data from the replica away and let it re-replicate completely with the same result. The empty brick was up for 1 minute and  3000 files appeared on it, which i find is the main cause of that saturation - gluster starts a blocking healing of too many files at once. I tried to limit the selfheal-count on volumes without success. Can anyone give
07:06 koubas some advice please?
07:08 jtux joined #gluster
07:18 puebele1 joined #gluster
07:24 jtux joined #gluster
07:24 6JTAAGRQV joined #gluster
07:31 stickyboy joined #gluster
07:32 ngoswami joined #gluster
07:36 ctria joined #gluster
07:40 ethemar joined #gluster
07:43 rb2k joined #gluster
07:53 sgowda joined #gluster
07:56 vshankar joined #gluster
07:59 spider_fingers joined #gluster
08:01 rotbeard joined #gluster
08:02 purpleidea joined #gluster
08:02 guigui1 joined #gluster
08:09 rgustafs joined #gluster
08:10 pai joined #gluster
08:10 vpshastry joined #gluster
08:32 lanning joined #gluster
08:36 puebele1 left #gluster
08:36 jtux joined #gluster
08:40 puebele2 joined #gluster
08:40 puebele2 left #gluster
08:44 mtanner_w joined #gluster
08:48 red-solar joined #gluster
08:51 glusterbot New news from newglusterbugs: [Bug 956564] glusterd stop returns 1 when successful <http://goo.gl/beCzM>
08:54 raghu joined #gluster
09:04 dobber_ joined #gluster
09:06 rastar joined #gluster
09:19 rastar joined #gluster
09:23 rastar1 joined #gluster
09:30 ujjain joined #gluster
09:35 rastar joined #gluster
09:47 guigui3 joined #gluster
09:52 glusterbot New news from newglusterbugs: [Bug 955588] Gluster volume info should return server uuid along with server ip and brick details. <http://goo.gl/ir68N>
09:54 drumfire joined #gluster
09:54 drumfire hey
09:54 drumfire i can hardly find info on IPv6, and everything I find is either obsolete or not working
09:54 drumfire I want gluster to work via IPv6 (only)
09:56 drumfire how can i get that to work?
09:58 rastar joined #gluster
10:00 drumfire joined #gluster
10:06 sgowda joined #gluster
10:11 rastar1 joined #gluster
10:12 rastar joined #gluster
10:19 bulde joined #gluster
10:22 vpshastry1 joined #gluster
10:23 hagarth joined #gluster
10:39 manik joined #gluster
10:42 karoshi joined #gluster
10:42 duerF joined #gluster
10:47 edward1 joined #gluster
10:50 yinyin joined #gluster
11:00 drumfire joined #gluster
11:05 kkeithley1 joined #gluster
11:05 kkeithley1 left #gluster
11:06 kkeithley1 joined #gluster
11:20 mtanner_ joined #gluster
11:21 H__ How does one fix such a broken files in glusterfs-3.3.1 ? http://fpaste.org/fPgy/
11:21 glusterbot Title: Viewing Paste #294502 (at fpaste.org)
11:21 avishwan joined #gluster
11:22 mohankumar__ joined #gluster
11:24 xymox joined #gluster
11:24 MinhP joined #gluster
11:24 Rydekull_ joined #gluster
11:27 bulde joined #gluster
11:28 atrius_ joined #gluster
11:28 soukihei_ joined #gluster
11:30 gmcwhistler joined #gluster
11:31 hagarth joined #gluster
11:31 piotrektt_ joined #gluster
11:34 77CAA0IP0 joined #gluster
11:34 ricky-ticky joined #gluster
11:37 avishwan left #gluster
11:37 aravindavk joined #gluster
11:46 Rydekull joined #gluster
11:55 balunasj joined #gluster
12:00 drumfire joined #gluster
12:05 dustint joined #gluster
12:21 sgowda joined #gluster
12:29 nueces joined #gluster
12:33 bennyturns joined #gluster
12:50 yongtaof joined #gluster
13:00 drumfire joined #gluster
13:12 nickw joined #gluster
13:13 vpshastry1 joined #gluster
13:17 jskinner_ joined #gluster
13:23 rwheeler joined #gluster
13:34 karoshi I think I found the reason for the client termination while running find I described two days ago. This happens when the brick I'm rebooting is the mount server. I thought this should not heppen. Is this expected?
13:38 lh joined #gluster
13:38 lh joined #gluster
13:38 berkc joined #gluster
13:38 spider_fingers hi guys, what is a possible reason for glusterd listening only on 24007?
13:38 guigui1 joined #gluster
13:39 spider_fingers i suppose it should be 24007 and 24008 both
13:39 ndevos ~ports | spider_fingers
13:39 glusterbot spider_fingers: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
13:41 spider_fingers yeah, thanks but i know that. so i have nodes properly listening on 24007. i can even telnet this port and get an error from remote "glusterd -debug" output, but still can't probe a peer
13:42 tjikkun_work joined #gluster
13:43 itisravi joined #gluster
13:46 spider_fingers any ideas?
13:46 drumfire so, IPv6 support, am I right to think it's broken?
13:49 vpshastry1 joined #gluster
13:49 ndevos spider_fingers: the source port of the initiating connection should be < 1024 iirc
13:50 ndevos there is an option you can set in the glusterd.vol file to accept other connections too
13:50 ndevos rpc.allow-insecure or something like that
13:51 ndevos spider_fingers: see https://lists.gnu.org/archive/html/​gluster-devel/2012-12/msg00031.html
13:51 glusterbot <http://goo.gl/mWWcv> (at lists.gnu.org)
13:55 manik joined #gluster
13:55 rcheleguini joined #gluster
13:55 spider_fingers but i dont' have a volume yet
13:56 spider_fingers i'd just set up gluster daemons, started them and now probing them
13:57 semiosis spider_fingers: check your glusterd log file for more information, /var/log/glusterfs/etc-glusterfs-glusterd.log
13:57 semiosis shoudl have some indication of why the probe failed
13:57 spider_fingers i was running glusterd -debug
13:57 ndevos glusterd --debug on both servers?
13:58 spider_fingers no connection in stdout, but if i telnet 24007 port and post smth like START, it bounces back rong command
13:59 semiosis karoshi: signs indicate your client was not connected to all replica bricks.  i'd explore that avenue to the fullest if i were you.
13:59 spider_fingers dammit, now i suspect the daemon from the 1st server, bbl
13:59 vpshastry joined #gluster
14:00 drumfire joined #gluster
14:00 karoshi semiosis: no, it definitely was
14:00 karoshi it said so in the logs
14:01 karoshi and I saw the ESTABLISHED connection to both
14:01 semiosis when you created new files, did they appear on all the bricks they should appear on?
14:01 karoshi doing a few tests with a floating mount server IP (that moves to the other brick when one goes away), the client has worked fine
14:01 karoshi yes, they did
14:02 semiosis i guess this is a bug then
14:02 semiosis 3.3.2?
14:02 karoshi not what I expected, I thought mount server was only relevant at mount time, but not a big deal
14:02 karoshi 3.3.1
14:02 karoshi I have the logs
14:02 karoshi 3.3.1 on debian
14:02 semiosis ok i'll try to replicate this issue on my test cluster
14:02 karoshi that'd be wonderful
14:03 semiosis probably wont be until later today though
14:03 karoshi basically, two-brick replicated vol, mounted on the client with the actual name of one of the bricks
14:03 karoshi once ounted, I did (on the client) find /mount/point
14:03 karoshi and let it go
14:03 semiosis karoshi: that's kinda vague, could you ,,(pasteinfo) and include the full client mount command (or fstab line) you're using?
14:03 glusterbot karoshi: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
14:04 karoshi since there are a lot of files, it takes about 30 minutes
14:04 karoshi sure
14:06 karoshi http://dpaste.org/QWEb2/
14:06 glusterbot Title: dpaste.de: Snippet #225793 (at dpaste.org)
14:06 karoshi if you need more info, just ask
14:07 semiosis thats great, thanks
14:08 karoshi to you!
14:09 deepakcs joined #gluster
14:10 bala1 joined #gluster
14:10 drumfire left #gluster
14:11 jclift_ joined #gluster
14:12 nickw hello, does anyone know under what circumstance will the 'gluster volume heal' command generate only the gfid instead of the actual filename?
14:14 shylesh joined #gluster
14:15 manik joined #gluster
14:16 spider_fingers i found the problem i mentioned above
14:18 shapemaker joined #gluster
14:19 spider_fingers you can't put rules like "iptables -A INPUT -i eth0 -p tcp --dport 24007 -j ACCEPT"
14:20 spider_fingers if you set "-i eth0", you get a problem. i wonder why
14:20 spider_fingers and eth0 is the only interface in the systems in my case
14:37 semiosis spider_fingers: i can put that rule in fine on my workstation
14:37 manik joined #gluster
14:37 spider_fingers yeah, that's odd
14:39 nickw SELinux?
14:39 nickw try to stop this if it's up
14:41 spider_fingers already done that, enforcing is not the matter
14:44 ndevos spider_fingers: the gluster command connects to glusterd over localhost (the "lo" interface)
14:46 saurabh joined #gluster
14:47 bugs_ joined #gluster
14:47 spider_fingers ndevos: but the traffic within kernel is not firewalled
14:47 ndevos within the kernel? the gluster command is userspace, and glusterd is just an other userspace process
14:48 aravindavk joined #gluster
14:49 ndevos spider_fingers: you could test the theory of needing localhost/lo by trying 'gluster --remote-host=$real_localhost_ip peer probe ...'
14:49 ndevos where $real_localhost_ip is an address of localhost, but not 127.0.0.1
14:50 guigui1 joined #gluster
14:52 premera joined #gluster
14:52 wN joined #gluster
14:53 rastar joined #gluster
14:53 spider_fingers ndevos: what is a real addr of localhost other than 127.0.0.1?)
14:53 ndevos spider_fingers: the ip of eth0
14:53 spider_fingers that's not localhost
14:54 ndevos well, its not a remote host either
14:54 ndevos s/$real)localhost_ip/$this_server_eth0_ip/
14:54 glusterbot ndevos: Error: '/^(?!s([^A-Za-z0-9\\}\\]\\)\\>\\{\\[\\(\\<​\\\\]).+\\1.+[ig]*).*$real)localhost_ip.*/' is not a valid regular expression.
14:54 ndevos s/$real_localhost_ip/$this_server_eth0_ip/
14:54 glusterbot ndevos: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
14:55 * ndevos counts messages
14:55 spider_fingers why the hell this bot thinks any regexes is for him?)
14:55 ndevos s/real_localhost_ip/this_server_eth0_ip/
14:55 glusterbot What ndevos meant to say was: where $this_server_eth0_ip is an address of localhost, but not 127.0.0.1
14:55 ndevos dammit, so close
14:55 ndevos s/real_localhost_ip peer/this_server_eth0_ip peer/
14:55 glusterbot What ndevos meant to say was: spider_fingers: you could test the theory of needing localhost/lo by trying 'gluster --remote-host=$this_server_eth0_ip peer probe ...'
14:56 spider_fingers huh)
14:56 ndevos that will make the gluster command to connect to $this_server_eth0_ip instead of 127.0.0.1
14:56 spider_fingers i'll check on this next storage
14:57 semiosis JoeJulian: glusterbot's s/// feature isn't working lately, 'case you hadn't noticed
14:57 ndevos the firewall may block access to lo/127.0.0.1, but allow access to glusterd on eth0
14:58 ndevos semiosis: yeah, dont use $ in glusterbot substitute :)
14:58 semiosis that, and the epic lag between a message and its being available for replacement
14:58 semiosis s/epic/huge/
14:58 spider_fingers so here's another question: what a client with glusterfs-fuse should have to mount dht volume on remote servers natively?
14:58 glusterbot What semiosis meant to say was: that, and the huge lag between a message and its being available for replacement
14:58 semiosis hey!
14:58 semiosis no lag that time
14:59 spider_fingers i think it shouldn't be running gluterd?
15:00 semiosis spider_fingers: fuse client just needs the glusterfs executable, maybe some shared libraries, no daemons running
15:01 spider_fingers okay then, but the client is unable to fetch volume
15:02 semiosis spider_fingers: logs
15:02 lpabon joined #gluster
15:02 bdperkin joined #gluster
15:02 aliguori joined #gluster
15:02 spider_fingers http://paste.fedoraproject.org/8662/66902156/ not so informative
15:03 glusterbot Title: #8662 Fedora Project Pastebin (at paste.fedoraproject.org)
15:04 semiosis is your server running the same version of glusterfs as your client (3.3.1) ?
15:04 spider_fingers nope, the servers running the older one
15:04 semiosis be more specific, what version is the server running?
15:05 spider_fingers glusterfs 3.2.7 built on Jun 11 2012 13:22:28
15:05 semiosis thats not going to work
15:05 semiosis incompatible
15:05 spider_fingers damn, thank you
15:05 semiosis 3.2 and 3.3 do not interoperate
15:05 semiosis yw
15:07 nickw does anyone know under what circumstance will the 'gluster volume heal' command generate only the gfid instead of the actual filename?
15:07 wN joined #gluster
15:09 theron joined #gluster
15:10 * semiosis doesn't
15:10 nickw thanks anyway
15:10 semiosis stick around, someone might
15:11 H__ my targetted brick rebalance worked
15:12 daMaestro joined #gluster
15:12 ndevos nickw: I think that happens when the filename of the gfid is not cached, but it definitely happens when the file has been removed and the entry under .glusterfs is still available
15:14 nickw how can i know whether the file is still there or not?
15:14 guigui4 joined #gluster
15:14 nickw since i don't know the actual file name
15:14 ndevos nickw: you can check if the entry under the .glusterfs directory points to a directory (symlink) or a file
15:14 semiosis ,,(gfid resolver)
15:14 glusterbot https://gist.github.com/4392640
15:14 semiosis nickw: ^^
15:14 semiosis roughly
15:15 ndevos if it is a file, you can check the inode number with 'ls -i $gfid'
15:15 nickw marked
15:16 ndevos ah, that uses 'find -inum $inode' too :)
15:16 neofob joined #gluster
15:16 semiosis ndevos: yep, hopefully saves people some keystrokes
15:17 ndevos semiosis: you can use 'stat --format=%i $GFIDPATH' to get the inode, that might be quicker
15:17 semiosis can you send me a pull request for a gist? :)
15:17 guigui1 joined #gluster
15:17 semiosis not sure if github supports that, but it would be sweet
15:18 ndevos I just wondered about that too :)
15:18 semiosis a comment would be good too, i can add one if you dont want to
15:19 semiosis so i can get back to it
15:21 vpshastry left #gluster
15:21 nickw thank you semiosis & ndevos
15:21 semiosis yw
15:21 ndevos semiosis: I've foked it, but cant see a create pull request option: https://gist.github.com/nixpanic/5460521
15:21 nickw i need to dig into it
15:21 glusterbot Title: gfid-resolver.sh (at gist.github.com)
15:21 ndevos nickw: good luck!
15:21 nickw thx
15:25 spider_fingers is there a repo with 3.3.x rpms for centos 6.x?
15:26 spider_fingers or you'd suggest it's better to build them
15:27 nickw ndevos: you are right, the file is gone but the entry is still there in .glusterfs, so how should i deal with the entry? delete it as well as the pre-dirs(just after the brick path)?
15:28 ndevos ~yum repo | spider_fingers
15:28 glusterbot spider_fingers: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
15:28 spider_fingers thx is it trusted?
15:29 spider_fingers oh he's here himself))
15:29 koubas joined #gluster
15:29 spider_fingers thx)
15:30 ndevos he's one of the main developers, I'd trust him ;)
15:32 spider_fingers i found another one: http://download.gluster.org/pub/gluster/glu​sterfs/3.3/3.3.1/CentOS/glusterfs-epel.repo
15:32 glusterbot <http://goo.gl/Y70il> (at download.gluster.org)
15:32 spider_fingers what is this?
15:34 spider_fingers left #gluster
15:34 jbrooks joined #gluster
15:38 koubas hi, can anyone help me, please? http://irclog.perlgeek.de/g​luster/2013-04-25#i_6745113
15:38 glusterbot <http://goo.gl/d8KKB> (at irclog.perlgeek.de)
15:39 nickw ndevos: i deleted the gfid entry from both of the bricks(replica 2) but the gfid is still shown in the 'heal info' command.. any idea how to fix this?
15:40 ndevos nickw: the heal-info command only updates itself every now and then, I think that you just have to wait some minutes
15:41 nickw ok, so no further actions needed except wait?
15:44 aliguori joined #gluster
15:47 zaitcev joined #gluster
15:48 jack_ joined #gluster
15:48 koubas joined #gluster
15:48 ujjain2 joined #gluster
15:53 glusterbot New news from newglusterbugs: [Bug 956781] On volume rebalance, max file limit is reached because files doesn't seem to be closed after the move (too many open files) <http://goo.gl/aWYUQ>
15:57 andrewjsledge joined #gluster
15:58 nickw ndevos: the gfid is disappeared, thx a lot. :)
15:58 ndevos nickw: nice to hear :)
16:06 hchiramm_ joined #gluster
16:13 gdavis33 joined #gluster
16:14 gdavis33 anyone having good luck with geo-replication features?
16:18 gdavis33 anyone having bad luck with geo-replication features?
16:25 _pol joined #gluster
16:28 slabgrha joined #gluster
16:28 andrewjs1edge joined #gluster
16:29 slabgrha I'm just testing gluster out - two peers - one replicated volume - while creating a bunch of files in the volume on a client, i shut off the network interface of the second peer - when i brought the interface up it's totally out of sync
16:30 slabgrha ran the stat command on the client - now there are files on the second peer, but a lot of the files are 0 bytes in length, whereas on the first peer the files have the correct data
16:31 slabgrha Am i doing something wrong?
16:33 Mo_ joined #gluster
16:34 itisravi joined #gluster
16:46 semiosis slabgrha: no you're not
16:46 semiosis what version of glusterfs are you using?
16:46 hagarth joined #gluster
16:47 portante` joined #gluster
16:47 slabgrha 3.3.1.1.el5
16:48 semiosis ok
16:48 semiosis the healing should be going on in the background, those 0 length files should be filled in with data when the background heal catches up to them
16:49 portante` joined #gluster
16:49 elyograg joined #gluster
16:50 slabgrha semiosis oh ok... my assumption is that i would bring down the interface on node2 during the write operation... bring the interface back up on node2 later on and the volumes would sync automatically on node2 w/o me having to run the find /export/vol01 -print0 ... stuff
16:50 slabgrha is that assumption correct?
16:51 slabgrha doesn't seem very fault tolerant otherwise ;)
16:51 semiosis since 3.3.0 there is a self-heal daemon which proactively heals files
16:51 slabgrha ok
16:51 semiosis you shouldn't have to find/stat them, but you can heal them ahead of the shd by doing that
16:52 semiosis or so i've been told
16:52 slabgrha ah ok
16:52 slabgrha one other thing i noticed, when i take down node2's interface, the writes to the volume (which is mounted via node1) kinda 'wedge'
16:52 slabgrha until i bring up node2's interface
16:53 slabgrha errr
16:53 slabgrha check that
16:53 slabgrha i'm impatient
16:53 slabgrha ;)
16:53 slabgrha sorry
16:54 semiosis ,,(ping-timeout)
16:54 glusterbot The reason for the long (42 second) ping-timeout is because re-establishing fd's and locks can be a very expensive operation. Allowing a longer time to reestablish connections is logical, unless you have servers that frequently die.
16:54 slabgrha ;) thanks
16:56 slabgrha one last noob question - i have to mount the volume via node1 or node2... is there a way to load balance between the nodes for mounts?  perhaps a floating IP address?
16:56 gdavis33 are there any ways to increase the speed and reliability of geo-replication?
16:57 gdavis33 V'm running v 3.3.1
16:59 y4m4 joined #gluster
16:59 semiosis slabgrha: ,,(mount server)
16:59 glusterbot slabgrha: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
16:59 semiosis slabgrha: ,,(rrdns)
16:59 glusterbot slabgrha: You can use rrdns to allow failover for mounting your volume. See Joe's tutorial: http://goo.gl/ktI6p
17:00 kkeithley| slabgrha: two bricks/nodes, with replication? Writes always go to both, no matter what. Read requests go to both, no matter what, the actual read occurs from which ever brick replies first. What would you load balance?
17:00 kkeithley| I don't see that you said NFS anywhere, so I presume you're using native.
17:07 slabgrha kkeithley| I mounted the volumes using mount -t glusterfs ... that's not NFS?
17:08 kkeithley| Correct, that's not NFS. Thus my comment.
17:08 slabgrha oh dear
17:08 slabgrha ok :)
17:08 slabgrha rockin
17:08 slabgrha :)
17:08 sprachgenerator joined #gluster
17:08 slabgrha semiosis / kkeithley| thanks for your help!
17:09 kkeithley| If you use NFS, then your rrdns becomes interesting
17:15 y4m4_ joined #gluster
17:19 aravindavk joined #gluster
17:24 _pol joined #gluster
17:25 _pol joined #gluster
17:31 jclift_ Hmmm, does anyone know of a programmatic way of figuring out which tcp ports Gluster is using?
17:32 jclift_ i.e. so "Gluster" can be queried for ports being used, so other things can automatically open them up appropriately
17:32 jclift_ Without needing human involved to think about the ports
17:33 jclift_ Starting to think something like that will be needed for a "really proper" integration solution to OpenStack / PackStack / etc
17:33 sprachgenerator having some interesting performance running 3.4 rdma/ib - anyone else?
17:34 jclift_ sprachgenerator: I keep on breaking my volumes with 3.4 / 3.5 when using rdma / ib
17:34 jclift_ So, feels a bit more buggy than expected
17:34 sprachgenerator and by interesting I mean the tcp/ib works, rdma does not
17:34 jclift_ Doesn't work at all, or just not working well?
17:34 sprachgenerator just not working well
17:35 jclift_ Yeah, I had similar personally when trying quickly about 1.5 weeks ago.
17:35 sprachgenerator I was wondering what the odds of this landing were: http://review.gluster.org/#/c/149/
17:35 glusterbot Title: Gerrit Code Review (at review.gluster.org)
17:35 jclift_ But I was using latest head of git master, and I didn't have time to properly analyze wtf
17:35 sprachgenerator looks like many rdma changes
17:35 flrichar joined #gluster
17:36 jclift_ Yeah, I don't have the skill to review the code.  Haven't done any rdma coding before.
17:36 jclift_ If you don't get an answer here, ask on the gluster-dev list + ask that patch author I guess.
17:37 semiosis jclift_: glusterd clearly serves that information to clients over port 24007, but i would just scrape the process table for glusterfsd :)
17:37 semiosis bbl
17:38 jclift_ semiosis: Yeah, might have to.
17:38 semiosis jclift_: ohh wait
17:38 semiosis gluster volume status --xml
17:38 semiosis ok bbl
17:38 jclift_ Oooh.
17:38 jclift_ That sounds like it has potential. :)
17:38 jclift_ semiosis: thx
17:39 * jclift_ hacks together new gluster command
17:39 jclift_ sudo gluster whichfuckingportsareweusing --today
17:39 * jclift_ gets back to real work
17:40 jclift_ Sometimes, the fact that this channel is logged escapes my memory
17:40 * jclift_ wishes he cared more :)
18:01 lpabon_ joined #gluster
18:02 pvivek joined #gluster
18:04 nueces joined #gluster
18:08 Oneiroi joined #gluster
18:08 rb2k joined #gluster
18:12 partner logged or not, you're not alone here ;)
18:23 slabgrha when i attempt to create a file on the mount, the client disconnects (http://pastebin.com/dYPU79Rt) - using a striped-replicate
18:23 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
18:24 slabgrha oops sorry
18:24 atrius_ joined #gluster
18:25 slabgrha (repaste of mnt.log: http://dpaste.org/LXV6R/)
18:25 glusterbot Title: dpaste.de: Snippet #225834 (at dpaste.org)
18:27 stickyboy joined #gluster
18:39 slabgrha are striped-replicated volumes GA?
18:40 semiosis slabgrha: yes, since 3.3.0, but i wouldnt recommend ,,(stripe) unless you've exhausted all other options
18:40 glusterbot slabgrha: Please see http://goo.gl/5ohqd about stripe volumes.
18:41 kkeithley| I don't suppose that the client disconnecting is related to using -stripe though.
18:42 andrewjs1edge joined #gluster
18:43 slabgrha yeah, i was looking through the errors in the mount.log  - saw some things like "Server and Client lk-version numbers are not same"
18:43 glusterbot slabgrha: This is normal behavior and can safely be ignored.
18:43 semiosis kkeithley|: failed to get stripe size is the last message before the crash in that log
18:44 slabgrha hmm
18:44 semiosis no idea what that error means but i've seen it a few times lately from people trying to use stripe
18:45 slabgrha oh.... ok... I was thinking RAID 1+0 ;)
18:46 slabgrha we're going to have a ton of small image files... maybe some larger log files
18:47 semiosis slabgrha: distributed-replicated
18:47 slabgrha maybe 4 servers with a brick of maybe 100G each
18:47 y4m4 joined #gluster
18:48 slabgrha oh.. interesting
18:49 slabgrha so, the files in the volume get balanced between replicated sets
18:49 stickyboy I've got a Java X11 program that runs really slow when homes are on network (nfs or gluster)... anyone have any pointers?
18:49 y4m4 joined #gluster
18:50 slabgrha semiosis - when i want to expand that storage, i would expand it in pairs then?
18:50 slabgrha (node01 node02) (node03 node04) (node05 node06) ...
18:50 semiosis stickyboy: here's some pointers: http://goo.gl/fkpJ9
18:51 stickyboy semiosis: Beautiful :P
18:51 semiosis :)
18:52 stickyboy I figured I'd ask here because lots of people use gluster in informatics, and therefore might have users who like crappy java apps :)
18:52 Rorik_ joined #gluster
18:53 semiosis slabgrha: yes thats right.  you can also expand the bricks themselves.
18:53 slabgrha sa-weet
18:53 slabgrha thanks!
18:53 semiosis slabgrha: rebalance is expensive, and i like to avoid it when possible
18:53 semiosis you need to rebalance after you add bricks/replica sets
18:53 slabgrha true
18:53 slabgrha aah
18:54 semiosis growing the underlying brick filesystems is easier imho
18:55 stickyboy semiosis: Wouldn't that depend on some sufficiently-advanced underlying storage?
18:55 Rocky_ joined #gluster
18:55 semiosis you can be fancy if you want to, but it's not necessary
18:56 semiosis simples would be, in a volume with replication, to replace one hdd with a larger one
18:57 semiosis more sophisticated options could use lvm to concat block devices together, zfs, maybe even some fancy hardware arrays
18:57 semiosis s/ples/plest/
18:57 glusterbot What semiosis meant to say was: simplest would be, in a volume with replication, to replace one hdd with a larger one
18:59 stickyboy semiosis: Ah. :)
19:00 jiffe98 joined #gluster
19:12 Supermathie Are there python bindings or something similar for glusterfs so I can connect as a client and issue specific operations (for testing)?
19:12 JoeJulian Son of a.... I migrated 60 bricks from their legacy ext4 lvs to new shiny xfs lvs night before last... or so I thought. Apparently I migrated 59. @%$@
19:12 cicero eek
19:12 JoeJulian @lucky glupy
19:12 cicero i mean that's still a 98.33% success rate
19:12 glusterbot JoeJulian: https://github.com/jdarcy/glupy
19:13 stickyboy Ahhh.
19:14 JoeJulian Supermathie: that glusterbot link is for you.
19:17 Supermathie JoeJulian: I don't think that's what I want, but I'm not sure... basically I'm trying to construct a test case for my setattr/chmod/truncate bug, so I want to be a client, not a translator.
19:17 Supermathie I'm working on porting nfsshell over to NFSv3 so I can use *that* at the moment...
19:17 semiosis Supermathie: a client is a particular kind of xlator stack, see a -fuse.vol file for example
19:18 semiosis as in /var/lib/glusterd/vols/foo/foo-fuse.vol or something like that
19:18 Supermathie mmmmm ok
19:18 Supermathie Ok, translating "client-side-operations" to gluster internal calls... yeah OK I get it.
19:19 stickyboy I see a lot of earlier GlusterFS guides have you writing vol files and stacking translators, but I never did that when I deployed 3.3.x... am I missing out?
19:19 JoeJulian jbrooks: I first read that Google+ post you made today as: "Drink the Cylon tea." Kinda give an completely different impression.
19:20 jbrooks JoeJulian, heh
19:21 semiosis stickyboy: manually editing volfiles has been deprecated since the glusterfs 3.1 release
19:22 JoeJulian but still... yes. You're missing out.
19:22 Supermathie JoeJulian: perhaps I'm taking the wrong approach and I should just write it as a C program... do similar little mini-programs already exist?
19:22 stickyboy semiosis: Good. Then I'm happy. :)
19:23 semiosis those pointers weren't enough to make you happy?!
19:24 JoeJulian Supermathie: jclift_ has been writing something using glupy. Perhaps he might be able to offer some insight if he's around.
19:25 Supermathie jclift_: Hello? :D
19:28 JoeJulian Supermathie: Make sure you run you test in trace mode and verify it against an strace. That might tell you more about where the problem is. If I'm remembering correctly, it's a sequence of events thing, right?
19:28 stickyboy semiosis: I'm happy I don't have to go edit my volfiles :)
19:29 stickyboy But I know users are going to complain about this Java app when I migrate their home dirs to Gluster next week
19:29 Supermathie JoeJulian: I've run it in trace mode, but there's soooooooooooooo much going on at the same time that it's impossible to watch like that. I do have a trace though and it behaves exactly as I suspected (chmod then truncate)
19:35 JoeJulian Well, I guess what I would do is to write a program/script/whatever that can create the problem in as few steps as possible. Then I'd run the trace in order to watch that problem pass through the translators.
19:35 JoeJulian That way you're not getting overwhelmed with information, but you can just watch the thing you want to watch.
19:35 Supermathie JoeJulian: That's my plan, yeah.
19:36 JoeJulian You should be able to do that through a mount.
19:36 JoeJulian Or I'm completely misunderstanding the problem. :/
19:39 Supermathie The trouble is that there's no normal semantics I can call to perform both operations in the same NFS SETATTR call
19:44 bulde joined #gluster
19:47 semiosis what normally causes the problem then?
19:47 semiosis is this oracle's fault?
19:48 semiosis :)
19:48 JoeJulian Yep, oracle's fault: "SETATTR is not guaranteed atomic. A failed SETATTR may partially change a file's attributes." - rfc1813
19:49 semiosis ha
19:49 JoeJulian Looking it up to see if I can duplicate it more simply though.
19:49 Supermathie That by itself isn't the problem. If a file starts off at 640 and you send a SETATTR(chmod(440),truncate(x)), the truncate should succeed, yes?
19:49 Supermathie GlusterFS ought to do the truncate first then the chmod
19:50 JoeJulian Seems logical, but not according to rfc. It doesn't seem like it would be all that hard to change gluster's behavior in that regard though.
19:51 Supermathie The RFC doesn't specify which operation should happen first - that's left up to the implementor. The implementor should take the most sensible action in this case - if we're removing permissions, do that last.
19:52 JoeJulian Yeah, I'm not arguing against that.
19:52 JoeJulian Just pointing out that oracle's "doing it wrong".
19:53 Supermathie Is ti?
19:53 Supermathie Is it?
19:53 Supermathie It's perfectly valid to send both requests in one setattr.
19:53 Supermathie And everybody else (NetApp, Linux, etc.) handles that.
19:54 JoeJulian Guess they should update the rfc then.
19:55 Supermathie It'd be more of a "Hey, when you're processing setattr request, think about these edge cases" which doesn't really fit into an RFC
19:55 JoeJulian And I wish someone would create an "accepted standards" instead of "request for comments"... seems like everyone's requesting comments forever and treating them as standards always bugs me a little.
19:56 Supermathie haha, yeah.
19:57 Supermathie They get converted into standards when Solaris gets shipped :D
19:58 _pol_ joined #gluster
19:59 _pol joined #gluster
19:59 JoeJulian So I think that you could fopen, fchmod, ftruncate then fsync and probably get the same behavior.
20:00 JoeJulian I would expect those transactions to get merged in the nfs client
20:01 Supermathie Right, but that's not the issue. Oracle's asking for fchmod, ftruncate in parallel.
20:01 Supermathie mmmmmm maybe
20:01 Supermathie lessee
20:01 JoeJulian Give it a whirl. I've got a couple errands to run. bbl.
20:08 lh joined #gluster
20:23 Supermathie JoeJulian: Nyet - the operations don't get combined
20:26 hchiramm_ joined #gluster
20:31 atrius_ joined #gluster
20:32 Rorik joined #gluster
20:33 nat joined #gluster
20:34 gorkhaan joined #gluster
20:37 gorkhaan Hi there! I was wondering if someone is using Gluster on Amazon Web Services. I would like to ask for a few tips about this. Today I have installed 3x2 instances ( two VM spread across Amazon Availability zone), using two Standard EBS drive with RAID0... the writing performance is poor. Any Ideas, or best practices would be helpful.
20:38 semiosis gorkhaan: i've had great success with glusterfs in ec2
20:38 semiosis how are you measuring performance?
20:39 gorkhaan dd if=/dev/zero of=test.img bs=1M count=10240
20:39 atrius_ joined #gluster
20:39 gorkhaan kill -USR1 `pidof dd`
20:39 gorkhaan the effective speed is around 1.4MB/s (megabyte)
20:40 gorkhaan iptraf shows 12MB/s replication data bandwith on the interfaces
20:40 Supermathie gorkhaan: try bs=64K or bs=128K
20:40 gorkhaan using ext4
20:41 gorkhaan how is your setup if I may ask?
20:41 gorkhaan I stacked up only two EBS drives into RAID0 is that enough? or should I try the EBS optimised stuff? or both?
20:42 semiosis well that all depends on what you need.  see ,,(canned ebs rant) for my thoughts on combining ebs volumes
20:42 glusterbot http://goo.gl/GJzYu
20:42 gorkhaan Supermathie, thanks, will try. However I have running wget to get an ubuntu image, that speed sucked too.
20:42 semiosis i use raw ebs volumes, not combined
20:42 semiosis but i have an embarrasingly parallel workload, so i'm not too concerned with single thread performance
20:42 semiosis ymmv
20:43 gorkhaan hm.
20:43 _pol_ joined #gluster
20:43 gorkhaan well I am running replication all over the 6 bricks, is that an overkill? what do you think?
20:43 semiosis raw ebs with xfs, that is, no lvm/mdadm in between
20:43 gorkhaan I see, I will definetly try that tomorrow.
20:43 semiosis gorkhaan: ,,(pasteinfo)
20:43 glusterbot gorkhaan: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
20:44 _pol joined #gluster
20:45 atrius_ joined #gluster
20:45 abyss^ joined #gluster
20:47 gorkhaan thanks for the tip.  what speed could you get out from it if I may ask?
20:48 semiosis i dont remember, it's been like two years since i did all that performance measurement & capacity planning
20:49 * semiosis should've written it down
20:49 semiosis tbh though i was more concerned with scaling out my application than how fast dd could write to a file
20:53 gorkhaan great. And thanks for that URL. What's your opinion about putting PostgreSQL data on gluster?  Around 16gigs database size.  not so frequently written/read, but there are times where spikes can occur.
20:54 semiosis no idea about that, i use rds
20:54 semiosis i think redshift has a postgres frontend
20:54 semiosis maybe someone else around here has experience with postgres on glusterfs
20:55 gorkhaan yeah, unfortunately there are some functions what are unavailable on Redshift. And the dev team... well you know them. :)
20:55 Supermathie gorkhaan: You may be interested in my Oracle on GlusterFS experiments :D
20:55 Supermathie It's running around 2000 TPS without tuning
20:56 gorkhaan sounds interesting. Should I be concerned about your smiley? ":D"    :)
20:57 atrius_ joined #gluster
20:58 gorkhaan If you have some doc, blog about that, I'll look into it. :)
21:06 H___ joined #gluster
21:08 atrius_ joined #gluster
21:10 twx_ joined #gluster
21:10 bfoster_ joined #gluster
21:10 jurrien__ joined #gluster
21:12 jmw2 joined #gluster
21:13 jmw2 left #gluster
21:15 H__ joined #gluster
21:21 Shdwdrgn joined #gluster
21:30 atrius- joined #gluster
21:38 rb2k joined #gluster
22:15 gdavis33 joined #gluster
22:15 Guest27955 joined #gluster
22:15 Keawman joined #gluster
22:15 genewitch joined #gluster
22:15 morse joined #gluster
22:15 flin_ joined #gluster
22:15 Dave2 joined #gluster
22:21 lh joined #gluster
22:27 zaitcev joined #gluster
22:32 lh joined #gluster
22:55 cfeller joined #gluster
23:40 fidevo joined #gluster
23:52 awheeler_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary