Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-13

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:09 rb2k_ joined #gluster
00:33 Hchl joined #gluster
00:39 tg2 hmm
00:39 tg2 doing a rebalance
00:39 tg2 localhost           241207         6.1TB        585404         38544    in progress
00:39 tg2 the 38544 is 'failures'
00:42 bulde joined #gluster
00:44 tg2 http://pastie.org/pastes/8037719​/text?key=qsdza108rl54nj16fuv6wq
00:44 glusterbot <http://goo.gl/UO7Mc> (at pastie.org)
00:44 tg2 i know for a fact there are files missing
00:46 kevein joined #gluster
00:47 tg2 3.3.1
00:54 Hchl joined #gluster
00:59 bala joined #gluster
01:02 tg2 is kernel 3.2.36 affected by the ext4 bug?
01:10 ccha joined #gluster
01:12 hagarth joined #gluster
01:19 majeff joined #gluster
01:24 tg2 nvm, bug was introduced in kernel 3.3.x so 3.2.36 is fine
01:24 bala joined #gluster
01:37 majeff joined #gluster
01:40 hjmangalam1 joined #gluster
01:43 Hchl joined #gluster
01:44 ccha joined #gluster
01:45 nightwalk joined #gluster
02:08 hjmangalam1 joined #gluster
02:15 majeff joined #gluster
02:40 puebele joined #gluster
02:59 vshankar joined #gluster
03:01 bharata joined #gluster
03:08 Hchl joined #gluster
03:17 mohankumar__ joined #gluster
03:20 mreamy joined #gluster
03:32 hjmangalam1 joined #gluster
04:04 rotbeard joined #gluster
04:21 miker joined #gluster
04:22 hajoucha joined #gluster
04:33 miker joined #gluster
04:36 CheRi joined #gluster
04:37 vpshastry joined #gluster
04:43 glusterbot New news from newglusterbugs: [Bug 973891] cp does not work from local fs to mounted gluster volume; <http://goo.gl/yKVT8>
04:43 aravindavk joined #gluster
04:46 45PAAN92X joined #gluster
04:50 vpshastry joined #gluster
04:55 Hchl joined #gluster
05:04 majeff joined #gluster
05:15 brosner joined #gluster
05:21 lalatenduM joined #gluster
05:25 satheesh joined #gluster
05:26 isomorphic joined #gluster
05:27 Hchl joined #gluster
05:32 hajoucha joined #gluster
05:33 bulde joined #gluster
05:37 ultrabizweb joined #gluster
05:38 psharma joined #gluster
05:40 shireesh joined #gluster
05:44 satheesh joined #gluster
05:54 raghu joined #gluster
05:55 isomorphic_ joined #gluster
06:04 jtux joined #gluster
06:05 ngoswami joined #gluster
06:07 arusso joined #gluster
06:09 Hchl joined #gluster
06:14 ctria joined #gluster
06:18 vimal joined #gluster
06:21 deepakcs joined #gluster
06:29 bharata joined #gluster
06:30 andreask joined #gluster
06:31 Hchl joined #gluster
06:42 ricky-ticky joined #gluster
06:55 mooperd joined #gluster
07:00 guigui3 joined #gluster
07:04 dobber_ joined #gluster
07:05 ekuric joined #gluster
07:19 hybrid512 joined #gluster
07:21 bala1 joined #gluster
07:27 ekuric joined #gluster
07:38 ujjain joined #gluster
07:57 majeff joined #gluster
08:09 kke left #gluster
08:30 sgowda joined #gluster
08:34 Hchl joined #gluster
08:39 mooperd joined #gluster
08:45 hchiramm_ joined #gluster
08:46 kevein joined #gluster
08:51 tziOm joined #gluster
09:01 Hchl joined #gluster
09:03 sgowda joined #gluster
09:05 ekuric joined #gluster
09:14 ramkrsna joined #gluster
09:29 rastar joined #gluster
09:38 mooperd joined #gluster
09:42 mooperd left #gluster
10:04 andreask joined #gluster
10:09 jbrooks joined #gluster
10:09 hajoucha https://bugzilla.redhat.com/show_bug.cgi?id=973891
10:09 glusterbot Bug 973891: urgent, unspecified, ---, csaba, NEW , cp does not work from local fs to mounted gluster volume;
10:11 realdannys1 joined #gluster
10:14 glusterbot New news from newglusterbugs: [Bug 952029] Allow an auxiliary mount which lets users access files using only gfids <http://goo.gl/x5z1R>
10:14 Hchl joined #gluster
10:16 pkoro joined #gluster
10:33 realdannys1 If anyone can help an EC2 user, stuck for days :( http://fpaste.org/18318/10750151/
10:33 glusterbot Title: #18318 Fedora Project Pastebin (at fpaste.org)
10:33 realdannys1 Full story here - http://serverfault.com/questions/515507/c​ant-mount-glusterfs-volume-on-ec2-client
10:33 glusterbot <http://goo.gl/et1xL> (at serverfault.com)
10:35 StarBeast joined #gluster
10:36 lalatenduM joined #gluster
10:38 ricky-ticky joined #gluster
10:48 Hchl joined #gluster
10:51 jbrooks realdannys1, you're mounting by the brick location when you should be mounting by the volume name
10:52 realdannys1 Really? I've just been doing exactly what all the tutorials say and that was one of them :/
10:52 realdannys1 so it should be url.com/gv0 ?
10:52 jbrooks realdannys1, well, now I'm doubting myself, but, I believe so
10:52 jbrooks Because that vol could have many bricks
10:53 realdannys1 but isn't the whole volume now at the folder on the server?
10:53 realdannys1 I'll try now
10:53 jbrooks What tutorial are you following?
10:53 jbrooks mount -t glusterfs HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR
10:53 jbrooks that's from http://gluster.org/community/documentation/ind​ex.php/Gluster_3.1:_Manually_Mounting_Volumes
10:53 glusterbot <http://goo.gl/cskQO> (at gluster.org)
10:54 realdannys1 thats the volume name mount directory - which is /export in my case
10:54 realdannys1 and it makes no difference i just tried, still fails
10:56 jbrooks you did mount -t glusterfs upload.themixtapesite.com:/gv0
10:56 realdannys1 yes
10:56 realdannys1 still failed :(
10:58 jbrooks and you can reach upload.themixtapesite.com from the client?
10:58 realdannys1 though look at my tutorial again, you are right - my bad - however it seems to be a connection problem
10:58 yinyin_ joined #gluster
10:58 realdannys1 well if not upload.themixtapesite.com I should be able to reach its elastic IP at least, which I've tried too and the public DNS none are working
11:00 realdannys1 I ever did a peer probe from my server to the client and it found it just fine
11:01 jbrooks I'd try sshing from client to server
11:01 jbrooks to test the connectivity
11:03 jbrooks realdannys1, this thread offers some advice to a similar-sounding issue: http://gluster.org/pipermail/glus​ter-users/2012-August/011142.html
11:03 glusterbot <http://goo.gl/uF9T7> (at gluster.org)
11:03 jbrooks no mention in there about the resolution -- it's probably something simple
11:05 realdannys1 any idea how I can SSH into the server from the client via command line?
11:06 Hchl joined #gluster
11:06 jbrooks assuming that ssh w/ password is disabled on your ec2 instance, you'd need to reenable it, this post talks about that: http://bharath.lohray.com/weblog/enabling​-root-access-over-ssh-to-an-ec2-instance/
11:06 glusterbot <http://goo.gl/SM778> (at bharath.lohray.com)
11:07 jbrooks and then, ssh root@upload.themixtapesite.com
11:07 jbrooks That's reenabling it on your server instance
11:07 jbrooks and sshing from your client
11:23 rastar joined #gluster
11:24 Hchl joined #gluster
11:26 chirino joined #gluster
11:34 yinyin_ joined #gluster
11:35 realdannys1 It'd be ok if I could find somewhere to put the pem file my server will actually access!
11:35 pkoro joined #gluster
11:42 realdannys1 ok finally, I can ssh in successfully from my client to the server @jbrooks - so now what? :/
11:48 charlescooke joined #gluster
11:49 bulde joined #gluster
11:53 rastar joined #gluster
12:03 DWSR joined #gluster
12:03 DWSR joined #gluster
12:10 bulde joined #gluster
12:15 Hchl joined #gluster
12:28 mohankumar__ joined #gluster
12:31 yinyin_ joined #gluster
12:33 Hchl joined #gluster
12:37 realdannys1 @semiosis
12:37 glusterbot realdannys1: I do not know about 'semiosis', but I do know about these similar topics: 'semiosis tutorial'
12:37 realdannys1 are you there?
12:37 realdannys1 Are you there @semiosis
12:37 realdannys1 An update to what I've tried - http://serverfault.com/questions/515507/c​ant-mount-glusterfs-volume-on-ec2-client
12:37 glusterbot <http://goo.gl/et1xL> (at serverfault.com)
12:38 realdannys1 I can SSH successfully from the client instance to the server instance - but it just won't bloody mount!
12:41 mooperd joined #gluster
12:56 aliguori joined #gluster
12:58 Hchl joined #gluster
13:18 Hchl joined #gluster
13:21 joelwallis joined #gluster
13:28 dewey joined #gluster
13:50 bambi23 joined #gluster
13:51 goerk_ joined #gluster
13:52 jag3773 joined #gluster
13:54 vpshastry joined #gluster
13:54 vpshastry left #gluster
13:59 majeff joined #gluster
14:04 dbruhn I am having a weird issue, it seems like the cluster is not synchronized in data for what is showing through the mount point
14:08 dbruhn from one client I am running some deletes, and getting the response that the directory is not empty
14:08 dbruhn but when I go to the directory it is empty
14:11 stickyboy exit
14:12 stickyboy Err. :P
14:12 bugs_ joined #gluster
14:15 realdannys1 http://serverfault.com/questions/515507/c​ant-mount-glusterfs-volume-on-ec2-client
14:15 glusterbot <http://goo.gl/et1xL> (at serverfault.com)
14:26 hagarth joined #gluster
14:27 georgeh|workstat joined #gluster
14:31 hybrid512 hi !
14:32 hybrid512 anyone using glusterfs with Debian Wheezy ?
14:33 hybrid512 I tried to add the Debian Wheezy repo for GlusterFS 3.3 and still, after apt-get update/upgrade, I'm still on GlusterFS 3.2 (which is in the base Debian repo) instead of 3.3 which should come from this repo ..
14:33 hybrid512 any Idea ?
14:33 Hchl joined #gluster
14:40 deepakcs joined #gluster
14:40 hagarth joined #gluster
14:41 Skunnyk hybrid512, what do you have in  your sources.list ?
14:45 realdannys1 joined #gluster
14:46 realdannys1 http://serverfault.com/questions/515507/c​ant-mount-glusterfs-volume-on-ec2-client @semiosis
14:46 glusterbot <http://goo.gl/et1xL> (at serverfault.com)
14:46 hybrid512 deb http://download.gluster.org/pub/gluster​/glusterfs/3.3/3.3.1/Debian/wheezy.repo wheezy main
14:46 glusterbot <http://goo.gl/iPHHm> (at download.gluster.org)
14:47 hybrid512 I imported the key without any problem
14:47 hybrid512 apt-get update ... no problem too
14:48 hybrid512 but when I do apt-cache show glusterfs-server, I get the information from the base repo
14:48 hybrid512 3.2
14:48 hybrid512 not the 3.3
14:48 dbruhn is there a way to enable log rotation from gluster itself or should I be using logrotate
14:52 gmcwhistler joined #gluster
14:52 Hchl joined #gluster
14:57 daMaestro joined #gluster
14:59 bulde joined #gluster
15:03 aliguori joined #gluster
15:09 zykure joined #gluster
15:13 Hchl joined #gluster
15:13 portante joined #gluster
15:15 bulde joined #gluster
15:19 dbruhn Also I am getting a directory not empty error even though the FS is not showing any files? what should I do
15:31 lalatenduM joined #gluster
15:33 Hchl joined #gluster
15:34 satheesh joined #gluster
15:37 jag3773 joined #gluster
15:45 jthorne joined #gluster
15:48 majeff joined #gluster
15:58 NcA^ anyone know what would cause gluster to spew out thousands of lines in the glustershd.log exactly every 10 minutes? It's all self heal info and managed to rack up 5Gb+ over the past few days
16:00 Hchl joined #gluster
16:25 saurabh joined #gluster
16:28 tziOm joined #gluster
16:39 Hchl joined #gluster
16:55 eightyeight why doesn't http://download.gluster.org/pub/​gluster/glusterfs/LATEST/Debian/ point to 3.4?
16:55 glusterbot <http://goo.gl/l2Ml1> (at download.gluster.org)
16:55 cicero perhaps it doesn't count beta
16:56 eightyeight ah. i see. shame on the debian developer for pushing non-stable software to sid. :)
16:57 eightyeight http://packages.debian.org/search?keyw​ords=glusterfs-server&amp;searchon=nam​es&amp;suite=unstable&amp;section=all
16:57 glusterbot <http://goo.gl/OuyvB> (at packages.debian.org)
16:57 bsaggy joined #gluster
17:09 aliguori joined #gluster
17:10 Hchl joined #gluster
17:14 cicero i wish i knew why mkdir -p was hanging :(
17:14 cicero mkdir("asdfasdftesting", 0755^C <unfinished ...>
17:14 cicero just sittin there
17:14 rb2k joined #gluster
17:16 tg2 # gluster volume rebalance storage fix-layout status
17:16 tg2 fails with: Usage: volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force]
17:17 tg2 incorrect usage message?
17:17 cicero volume name is storage?
17:17 tg2 yes
17:17 tg2 # gluster volume rebalance storage fix-layout start
17:17 tg2 Starting rebalance on volume storage has been successful
17:19 cicero weird, the source looks fine
17:19 tg2 yeah
17:20 cicero oh perhaps you can't use fix-layout status
17:20 cicero and need to use just status
17:20 cicero https://github.com/gluster/glusterfs/blo​b/master/cli/src/cli-cmd-parser.c#L2452
17:20 glusterbot <http://goo.gl/y37jp> (at github.com)
17:20 cicero starting from there
17:20 cicero yeah probably just status
17:21 cicero Usage: volume rebalance <VOLNAME> {fix-layout {start|stop}|status} [force]
17:21 cicero perhaps?
17:21 joelwallis left #gluster
17:22 bsaggy Hey guys, I've got a replicated cluster between two bricks. How long should it take for a file written to Brick1 to replicate to Brick2?
17:22 cicero should be near instantaneous in my experience
17:22 cicero but it all depends on your networking and kind of files
17:22 cicero (many small files vs few large files)
17:22 bsaggy Yea, that's what I thought.
17:22 cicero i've mostly dealt w/ the former
17:23 MrNaviPa_ joined #gluster
17:23 ctria joined #gluster
17:24 bsaggy So, gluster flaked out on the second brick. I killed it and restarted it, the cluster came back online, but it seems replication is slow, if it works at all.  Any suggestions?
17:25 cicero which version?
17:25 cicero 3.3 has self heal
17:25 cicero so you might need to kick that off
17:25 cicero i'm not entirely sure -- i'm actually in the same boat right now
17:25 cicero 2 brick repl
17:25 bsaggy I'm using 3.1.3
17:26 cicero ah
17:26 cicero http://gluster.org/community/documentation/index.p​hp/Gluster_3.1:_Triggering_Self-Heal_on_Replicate
17:26 glusterbot <http://goo.gl/pMJrJ> (at gluster.org)
17:26 cicero that jam, basically you may have to traverse the filesystem with stat
17:26 cicero and that'll resolve any inconsistencies
17:27 cicero supposedly
17:27 cicero so i'm not sure.
17:30 bsaggy Cool, I'll check out this link. Thanks cicero & glusterbot.
17:31 cicero gl
17:31 bsaggy Thanks.
17:32 failshell joined #gluster
17:34 Hchl joined #gluster
17:39 bsaggy So these commands should be run from a client which accesses the cluster, and not the gluster servers themselves?
17:43 realdannys1 joined #gluster
17:43 cicero yeah
17:43 cicero well, whatever speaks the gluster protocol
17:43 cicero to the servers
17:43 cicero but they're meant to be run against the FUSE mountpoint
17:43 cicero as opposed to the brick fs
17:43 tziOm Is any work beeing done on improving stat time/stat cache/.. in gluster?
17:44 cicero tziOm: dunno, but there are params you can tweak
17:44 cicero both at the gluster level and on the kernel level iirc
17:46 tziOm cicero, ok, can you point me in some direction. I have tried tweaking what I can find documented..
17:46 cicero then you probably know more than i do :P
17:46 cicero http://gluster.org/community/documentation/i​ndex.php/Gluster_3.2:_Setting_Volume_Options
17:46 glusterbot <http://goo.gl/dPFAf> (at gluster.org)
17:46 cicero that jam
17:47 tziOm compared to straight nfs, gluster is approx 20 times slower (deliverting 100 small files in a loop)
17:47 cicero yeah, but nfs is a single point of failure, right?
17:47 tziOm ah.. those options seems to do nothing.
17:47 tziOm cicero, could be, depending on design
17:47 stickyboy gluster's fuse mount point is faster if you have many nodes.
17:48 stickyboy You can see massive speedups with NFS if you have few nodes.
17:48 tziOm stickyboy, yeah.. but nfs has some caching benefits
17:48 stickyboy tziOm: Also because with NFS, replication is done on the server side.
17:48 tziOm stickyboy, sure, but writes is no problem, this is read same files over and over usage
17:49 tziOm webserving
17:49 cicero the strategy i've adapted,
17:49 stickyboy tziOm: Ok.
17:49 cicero which may defeat the purpose for you, i'm not sure
17:49 cicero is to rsync to a local filesystem
17:49 cicero but then again my usage pattern is related to CI builds, not web content
17:49 stickyboy tziOm: What interconnect are you using?
17:49 tziOm its gbe now
17:50 tziOm I could do rdma, but would like to see some better numbers first.
17:50 tziOm because these numbers are not network latency (only)
17:51 stickyboy tziOm: Yeah, I'm using 1GbE as well.  I was getting ~80-90MB/sec with NFS.
17:52 stickyboy Now 50 or so with FUSE, as we're doing replica 2, so the client writes to both nodes at the same time.
17:52 tziOm I dont think its big file speed that is the (main) problem with the gluster performance I am seeing..
17:52 tziOm not that gluster advertises great small file performance.. but still.. 20x time is a... lot
17:52 tg2 # gluster volume rebalance storage migrate-data start
17:52 tg2 Usage: volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force]
17:52 tg2 sigh
17:52 stickyboy tziOm: Ah, we're interested in larger files; not CERN Physics large... but big-ish.
17:52 tg2 http://gluster.org/community/document​ation/index.php/Gluster_3.2:_Rebalanc​ing_Volume_to_Migrate_Existing_Data
17:52 glusterbot <http://goo.gl/4sqgt> (at gluster.org)
17:53 tziOm stickyboy, what is the main use for these large files?
17:53 vpshastry joined #gluster
17:53 stickyboy tziOm: Bioinformatics
17:53 stickyboy Genomes, etc.
17:53 tziOm ok.. I'm in hosting (web/mail/..)
17:53 stickyboy Ah
17:54 stickyboy Well JoeJulian is doing web stuff too.  He's written some articles on his blog about it.
17:54 stickyboy You might have a look there:  http://joejulian.name/blog/category/glusterfs/
17:55 hagarth joined #gluster
17:55 stickyboy He does sysadmin for a large web retailer, so I think he's had to address issues with PHP performance
17:55 tg2 php definitely faster off of nfs
17:55 tg2 if you have a repo for example
17:55 tg2 on gluster
17:55 tg2 and have all yoru webservers mounted to it
17:55 stickyboy tziOm: http://joejulian.name/blog/nfs-mount-for-glusterf​s-gives-better-read-performance-for-small-files/
17:55 glusterbot <http://goo.gl/5IS4e> (at joejulian.name)
17:56 tg2 for some reason php issues a stat() command when doing an include
17:56 tziOm everything is stat
17:57 tziOm nfs does stat
17:57 zaitcev joined #gluster
17:57 tg2 for some reason its much faster in nfs
17:57 tziOm stickyboy, been there..
17:57 tg2 http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/
17:57 glusterbot <http://goo.gl/uDFgg> (at joejulian.name)
17:58 stickyboy tziOm: Read all his posts :)
17:58 stickyboy Become wise :)
17:58 tziOm tg2 -  think I have been there too.. problem is me and joejulian have different perspectives on performance.. ;)
17:58 tg2 i notice that even with 6 bricks
17:58 tg2 nfs is faster for most things
17:58 tg2 including single file transfers (move/copy operations)
17:58 tg2 on 2gb+ files
17:59 tg2 but there are some issues iwth nfs we saw
17:59 tg2 like dropping symlinks
18:00 tg2 and files not being available after the write operation finishes in some instances
18:01 tg2 getting .nfsxxxx in place of your file
18:01 tg2 etc
18:01 stickyboy tg2: Yikes
18:01 tg2 yeah
18:01 tg2 so fuse is a bit slower
18:01 stickyboy tg2: We dropped NFS because we needed locking support.
18:01 tg2 but more stable
18:02 stickyboy But yeah, I'm happier using FUSE anyways.
18:02 tg2 yeah if you have a file open by another process and you move it or delete it (via nfs)
18:02 tg2 that causes some really weird issues
18:02 tg2 but I do wish fuse performance was higher for small files
18:02 tg2 on 10gbe
18:02 tg2 getting 60-70MB/s on single file transfer
18:02 tg2 when the underlying disks on each brick can do 700+
18:03 stickyboy tg2: I hear that.  I'm running on top of some decent hardware 12-disk RAID5 arrays, so the 50-60MB/sec is kinda annoying. :P
18:03 tg2 my r5 arrays have 11 4Tb ultrastars
18:03 tg2 and I have 6 arrays
18:03 stickyboy But I doubt my users even notice, to be honest.
18:03 tg2 each one can do 600+MB/s sustained read/write
18:04 tg2 yet reading a file through the fuse mount is like painfully slow
18:04 tg2 but concurrency is decent
18:04 tg2 remove-brick and add-brick are nice features but remove-brick is buggy as I've seen, our bricks were accpeting files while in remove-brick status.
18:05 stickyboy tg2: Eek.
18:05 tg2 if you have a vm disk on your brick
18:05 tg2 that you're rebalancing
18:05 stickyboy I wouldn't want to be relying on remove-brick... I'd probably do it on a weekend if I had to :P
18:05 tg2 that messes it up pretty spectacularly
18:05 tg2 ;)
18:05 tg2 i'm guessing it does a snapshot, moves it, then deletes the old one
18:05 tg2 so anything written in the interim is gone
18:06 tg2 would be cool to have copy-on-write awareness
18:06 mooperd joined #gluster
18:07 tg2 would be a complicated diff process to do it
18:08 tg2 but not impossible
18:08 tg2 that way you could be writing a file while its rebalancing and it would be transparent
18:08 Hchl joined #gluster
18:08 tg2 its how vmware does it when you do a disk migration live
18:09 tg2 there's probably code for it in RHEV since it also supports live migration of vms.
18:14 ferringb tg2: look into 3.4... that "remove-brick but still accepting" sounds like a change they made to migrate open fd's to new graphs
18:14 ferringb first thought, w/out that sort of functionality, a remove-brick wouldn't work till all open access to that file had finished
18:16 andreask joined #gluster
18:17 vpshastry left #gluster
18:20 joelwallis joined #gluster
18:20 ollivera joined #gluster
18:26 vpshastry joined #gluster
18:29 realdannys1 Just stopped and started my Ec2 instance and the public DNS has changed @semiosis, so the only thing that stays consistent is the elastic IP - but I can't mount bricks with the elastic IP it just fails and reverse IPing it just gives me the public DNS I can already see in the panel which channels on stop/start...
18:31 joelwallis joined #gluster
18:32 realdannys1 Ah no, thats wrong - when you stop the elastic IP becomes disassociated and when you bring it back, the public DNS comes back again - my bad…I'm trying one last thing booting it so the server and the client share the same security group, only idea left. After that i'll have to look for alternatives to Gluster
18:43 y4m4 joined #gluster
18:57 joelwallis joined #gluster
19:02 semiosis realdannys1: back
19:03 realdannys1 hi! Still stuck with same issue - just tried doing a new AMI in the same security group as the client, but same issue
19:03 realdannys1 Im not sure this is going to work for me
19:03 realdannys1 I wonder if its a linux issue or something deeper that I don't know - but it SSH's fine so it could be a bug in Gluster
19:03 realdannys1 both instances are running Centos6.4
19:04 semiosis realdannys1: what version of gluster?
19:05 mooperd joined #gluster
19:07 realdannys1 3.3
19:08 semiosis realdannys1: should be fine
19:08 semiosis lets get back to basics
19:08 semiosis first of all, gluster does work on ec2, well
19:09 semiosis gotta be some simple issue we're overlooking
19:09 semiosis so...
19:09 semiosis did you install gluster on each server?  or install on one and then clone it?
19:09 semiosis also please ,,(pasteinfo)
19:09 glusterbot Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
19:10 realdannys1 installed on each - I only have one sever for testing
19:10 semiosis ok
19:10 realdannys1 so installed on one I should say
19:10 semiosis thats good
19:13 realdannys1 volume info - http://fpaste.org/18516/11507791/
19:13 glusterbot Title: #18516 Fedora Project Pastebin (at fpaste.org)
19:16 semiosis that looks good.  whats your mount command?
19:16 MrNaviPa_ joined #gluster
19:17 realdannys1 mount -t glusterfs http://ec2-107-20-245-144.c​ompute-1.amazonaws.com:/gv0 /var/www/vhosts/i-16852366/wp-content​/uploads/gravity_forms/1-9e5dc27086c8​b2fd2e48678e1f54f98c/2013/02/gluster
19:17 glusterbot <http://goo.gl/rHtrv> (at ec2-107-20-245-144.compute-1.amazonaws.com)
19:17 semiosis ok, so there's your problem (or one of them)
19:17 semiosis drop the http://
19:18 semiosis also you dont need the / before gv0
19:18 semiosis just ec2-107-20-245-144.compute-1.amazonaws.com:gv0
19:18 semiosis though the / should be ignored if its there
19:18 realdannys1 sorry I dunno where the http:/; came from I've not been using that
19:18 realdannys1 I will get rid of the slash though
19:19 tg2 ferringb: do you have a link to this change? i'm running 3.3.1 and had this bug, has it been fixed?
19:19 realdannys1 OK - just ran mount -t glusterfs ec2-107-20-245-144.compute-1.amazonaws.com:gv0 /var/www/vhosts/i-16852366/wp-content​/uploads/gravity_forms/1-9e5dc27086c8​b2fd2e48678e1f54f98c/2013/02/gluster
19:19 realdannys1 and same problem
19:19 ferringb tg2: look for 8946bd7d239dc505d53270e29684ccf9df9ce3a0
19:20 ferringb ah
19:20 ferringb tg2:  http://review.gluster.org/5167
19:20 glusterbot Title: Gerrit Code Review (at review.gluster.org)
19:20 semiosis realdannys1: can you please pastie/fpaste yoru client log file?  probably /var/log/glusterfs/var-www-vhosts-i-16.....log
19:20 ferringb tg2: haven't dug deeply into it- just saw it in the commits, and it caught my eye
19:21 realdannys1 semiosis: you can see it here - http://serverfault.com/questions/515507​/cant-mount-glusterfs-volume-on-ec2-cli​ent?noredirect=1#comment583861_515507
19:21 glusterbot <http://goo.gl/9xpxW> (at serverfault.com)
19:23 semiosis realdannys1: no route to host... that's unusual.  lets double check that.  please fpaste the output of 'dig ec2-107-20-245-144.compute-1.amazonaws.com' and 'route -en' on your client
19:23 semiosis s/dig/host/
19:23 glusterbot What semiosis meant to say was: realdannys1: no route to host... that's unusual.  lets double check that.  please fpaste the output of 'host ec2-107-20-245-144.compute-1.amazonaws.com' and 'route -en' on your client
19:23 semiosis if you prefer
19:26 realdannys1 Here you go - http://fpaste.org/18521/15155013/
19:26 glusterbot Title: #18521 Fedora Project Pastebin (at fpaste.org)
19:26 semiosis can you ping the server from the client?
19:29 realdannys1 I can SSH into it
19:29 realdannys1 Do you know what
19:29 realdannys1 I think I've just found it
19:29 realdannys1 well
19:29 realdannys1 I've found a problem, whats the Centos RPM for 3.3.1.1?
19:30 tg2 very nice
19:30 tg2 ok
19:30 tg2 so it has been fixed in 3.4
19:30 tg2 when is the ETA for release to stable on 3.4?
19:30 tg2 i see there are rpms and such already avaialble
19:32 partner inode link failed on the inode (00000000-0000-0000-0000-000000000000)
19:32 partner does that sound familiar to anybody?
19:34 partner i made replica 2 of a one brick dist volume and self-heal is filling logs with such lines
19:35 partner disk utilization is ~100% on the old server and load went 1 -> 8 while the newly added is doing only fraction of those numbers..
19:36 realdannys1 @rpm
19:36 glusterbot realdannys1: The official community glusterfs packges for RHEL/CentOS/SL (and Fedora 17 and earlier) are available here http://goo.gl/s077x
19:39 partner the volume is still accessible and seems to work, is probably a bit slow.. is there something i could do? debian wheezy, 3.3.1, etc..
19:39 semiosis realdannys1: ,,(yum repo)
19:39 glusterbot realdannys1: The official community glusterfs packages for RHEL (including CentOS, SL, etc), Fedora 17 and earlier, and Fedora 18 arm/armhfp are available at http://goo.gl/s077x. The official community glusterfs packages for Fedora 18 and later are in the Fedora yum updates repository.
19:39 semiosis realdannys1: what problem do you think you found?
19:41 dbruhn joined #gluster
19:41 semiosis brb
19:42 realdannys1 semiosis: based on a server fault post I came across with someone with same problem, turned out they had different versions of gluster on client and server - I just check and the server is running 3.2.2 - so I'm updating it now
19:42 realdannys1 …fingers crossed
19:43 ferringb partner: yeah, I've seen a lot of that
19:43 partner ferringb: nothing to worry about?
19:43 ferringb no clue, tbh, although I've been looking into it since the perf hasn't been particularly balanced
19:44 realdannys1 Ok, they've both on glusterfs-3.3.1-15 now
19:44 partner i did the operation to improve the performance.. thought of course it takes time to balance the files to both servers but the first one takes now huge hit compared to previous state
19:45 realdannys1 ARGH!!! it still failed!!!!
19:45 hagarth joined #gluster
19:45 realdannys1 this is mental!
19:45 tg2 http://pastie.org/pastes/8040201​/text?key=s6i24lh54syiqhafacefbg
19:45 glusterbot <http://goo.gl/IhK78> (at pastie.org)
19:46 tg2 any idea why rebalance is doing this?
19:46 ferringb partner: part of that hit is going to be the second rebuilding
19:46 ferringb this is assuming it's a replicate setup. if it isn't, then you need to rebalance (don't ask me of that behaviour, yet to experience it)
19:46 tg2 ok files are on my brick
19:46 tg2 but not in the volume
19:47 partner ferringb: its a replica yes (now, was just 1 brick dist)
19:47 bulde joined #gluster
19:48 partner i've done this once before but not for so heavily loaded clients/volume
19:48 realdannys1 :(
19:49 ferringb partner: things still staying up, or is it too loaded?
19:50 mafrac1 joined #gluster
19:50 partner ferringb: thats the good thing, stuff is still up and running, only that i have 40000+ errors in log and it keeps increasing fast so thought to ask what is going on :)
19:52 ferringb partner: check what is spitting it- self-heal-daemon I'd assume
19:52 ferringb either way, yeah, it does make noise during rebuild/balancing
19:52 ProT-0-TypE joined #gluster
19:54 partner tailing the log.. its only those lines endlessly..
19:55 partner [2013-06-13 22:54:19.236590] E [afr-self-heald.c:685:_link_inode_update_loc] 0-csrv-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)
19:55 partner and nothing else
19:56 mafrac1 Hi all. I don't find a solution for this:
19:56 mafrac1 # service glusterd start
19:56 mafrac1 Starting glusterd:                                         [  OK  ]
19:56 mafrac1 # service glusterd status
19:56 mafrac1 glusterd dead but subsys locked
19:56 bulde joined #gluster
19:56 mafrac1 Could you help me, please?
19:56 tg2 can anybody shed some light on why this rebalance isn't working?  this is the error i'm getting but the file exists in the brick and in the array...
19:56 tg2 http://pastie.org/pastes/8040201​/text?key=s6i24lh54syiqhafacefbg
19:56 glusterbot <http://goo.gl/IhK78> (at pastie.org)
19:58 tg2 I see this: http://gluster.org/pipermail/glu​ster-users/2012-June/010799.html
19:58 glusterbot <http://goo.gl/b6bm1> (at gluster.org)
19:59 tg2 avati mentions that that would be fixed however, was it, and if so why is it still showing in errors?
20:04 semiosis mafrac1: pastie.org the glusterd log file, /var/log/glusterfs/etc-glusterfs-glusterd.log
20:04 mafrac1 ok
20:05 mafrac1 There isn't a /var/log/glusterfs/etc-glusterfs-glusterd.log
20:06 mafrac1 could be /var/log/glusterfs/etc-glusterfs-glusterd.vol.log ?
20:08 mafrac1 semiosis: There isn't a /var/log/glusterfs/etc-glusterfs-glusterd.log. Could be /var/log/glusterfs/etc-glusterfs-glusterd.vol.log ?
20:09 semiosis yeah that
20:11 MrNaviPa_ joined #gluster
20:18 Hchl joined #gluster
20:18 hagarth joined #gluster
20:21 mafrac1 semiosis: Here you are: http://pastie.org/8040300
20:21 glusterbot Title: #8040300 - Pastie (at pastie.org)
20:22 semiosis mafrac1: how did you install glusterfs?  source?  package?
20:25 mafrac1 From repo http://download.gluster.org/pub/gluster/glu​sterfs/LATEST/EPEL.repo/glusterfs-epel.repo . the server was working some mounts and after last server restart gluster do it
20:25 glusterbot <http://goo.gl/5beCt> (at download.gluster.org)
20:26 mafrac1 the volume servidor4-backups is not used ye but I don't know how to remove now that is stoped
20:27 mafrac1 semiosis: another thing is that I added one peer one day before and I did some rebalances
20:28 mafrac1 the next day this brick was offline on one volume so I tried to restart it with this result
20:38 Hchl joined #gluster
20:39 partner hmph, a whopping 1.2 GB already self-healed after two hours
20:41 semiosis mafrac1: maybe you should try resetting the glusterd config on that server.  move all the files EXCEPT glusterd.info out of /var/lib/glusterd, then start glusterd
20:41 semiosis you'll need to probe to/from that server from another one, possibly restarting glusterd as well, then it should resync its config from a peer
20:41 semiosis i hope
20:45 bsaggy I have a node with a brick in which glusterfs is flaking out.  I stopped glusterfs and started it back up earlier today, the cluster came up and everything seemed fine. However, gluster has flaked out again only a few hours later. Seems gluster CPU usage jumps up, as does the CPU load average.
20:45 bsaggy Any ideas?
20:50 stat1x joined #gluster
20:56 semiosis bbl
21:21 mafrac1 semiosis: I did it
21:21 mafrac1 now, how do I resync all?
21:22 mafrac1 now there isn't volumes present
21:25 rb2k joined #gluster
21:28 mafrac1 semiosis: Thank you very much. I completed sync following http://gluster.org/community/documen​tation/index.php/Gluster_3.2:_Brick_​Restoration_-_Replace_Crashed_Server
21:28 glusterbot <http://goo.gl/60uJV> (at gluster.org)
21:34 rcoup joined #gluster
21:40 jag3773 joined #gluster
21:46 realdannys1 joined #gluster
21:53 jag3773 joined #gluster
21:55 realdannys1 semiosis: anymore ideas, changing the versions didn't do squat :(
21:59 primusinterpares joined #gluster
22:02 FilipeMaia joined #gluster
22:02 FilipeMaia Hi. I've just created a simple replicated volume with no problems. It mounted fine. But when I try to ls I see glusterfsd using a large amount of cpu and nothing happens (the ls should come out empty)
22:03 FilipeMaia I've used 32 bricks (maybe too much?)
22:04 mafrac1 left #gluster
22:04 FilipeMaia I seem to be getting an error like: [2013-06-14 00:01:29.333393] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1017)
22:04 glusterbot FilipeMaia: That's just a spurious message which can be safely ignored.
22:05 FilipeMaia Smart bot
22:07 FilipeMaia 5 minutes for the ls and counting
22:12 Hchl joined #gluster
22:16 y4m4 joined #gluster
22:17 FilipeMaia It seems that I can't do ls on the machine that issues the volume create
22:28 FilipeMaia I seem to have this issue https://gist.github.com/hogem/4043772
22:28 glusterbot Title: getdents loop on glusterfs distributed volume (ext4 filesystem) (at gist.github.com)
22:30 It_Burns joined #gluster
22:30 It_Burns wow, busy place, or just a gathering of lurkers
22:31 It_Burns Anywho, anyone have a moment to talk a bit about Gluster? I'm running a small HPC system and think Gluster may be a good fit, just have some questions
22:33 simpleirc1 joined #gluster
22:37 a2 FilipeMaia, are you using ext4?
22:37 FilipeMaia Yes
22:37 a2 FilipeMaia, upgrade to 3.3.2 at least
22:37 a2 It_Burns ?
22:38 FilipeMaia a2: Just the client, or the server also?
22:39 FilipeMaia It would be nice to have this more prominently in the docs. Ext4 is not exactly a rare fs
22:40 simpleirc1 joined #gluster
22:40 a2 FilipeMaia, just the client.. though it is good practice to keep server and client in same version
22:40 FilipeMaia a2: ok, thanks
22:41 FilipeMaia Where can I find packages for 3.3.2?
22:41 FilipeMaia I can only find 3.3.1
22:42 It_Burns a2: Well I guess my question is a bit complex. I have a 15 node cluster and SAN set aside for cluster storage. My OS is CentOS. I'm trying to figure out the best way to implement Gluster in a test instance to try it out.
22:43 It_Burns I'm debating if there's merit in having the SAN (iSCSI) volume presented to all nodes, then peering them together using gluster, or just have the iSCSI volume presented to a storage/IO node then use the gluster client on the compute nodes
22:44 ctria joined #gluster
22:45 badone joined #gluster
22:48 mooperd joined #gluster
22:49 semiosis @later tell mafrac1 glad to hear it
22:49 glusterbot semiosis: The operation succeeded.
22:49 a2 FilipeMaia, 3.3.2qa3 is the latest
22:49 FilipeMaia a2: not according to the front page
22:50 FilipeMaia Latest GA version of GlusterFS (ver. 3.3.1
22:50 a2 It_Burns, gluster cannot use a SAN any more intelligently than using it like direct attached drives
22:50 a2 FilipeMaia, http://bits.gluster.org/pub/glu​ster/glusterfs/3.3.2qa3/x86_64/
22:50 glusterbot <http://goo.gl/cHdrf> (at bits.gluster.org)
22:51 FilipeMaia Thanks
22:51 FilipeMaia I'm getting the following error when installing the RPMs http://paste.debian.net/10260/
22:51 glusterbot Title: debian Pastezone (at paste.debian.net)
22:51 a2 FilipeMaia, uh, ignore the geo-replication package
22:52 FilipeMaia a2: I didn't install it
22:52 chirino joined #gluster
22:52 FilipeMaia a2: just glusterfs, fuse and the server
22:52 a2 FilipeMaia, then you need to probably uninstall the old geo-replication package?
22:53 FilipeMaia a2: it was never installed
22:54 FilipeMaia i think
22:54 FilipeMaia a2: nope, not installed
22:57 a2 this file - usr/libexec/glusterfs/python/syncdaemon/gsyncd.py is part of glusterfs-geo-replication RPM
22:57 a2 if you are getting an error from that file, you have it on your system
22:57 It_Burns a2: Ah ok, thanks
22:58 k12admin joined #gluster
22:58 FilipeMaia a2: I do not have that file (or a /usr/libexec/glusterfs directory)
23:02 FilipeMaia Removing the RPMs and installing them again does not reproduce the error
23:03 FilipeMaia Now I have the issue that some bricks still think the volume is alive and others don't
23:03 FilipeMaia some nodes
23:03 Hchl joined #gluster
23:03 a2 did you upgrade all the servers?
23:04 FilipeMaia yes
23:04 a2 killall glusterd glusterfsd glusterfs on all nodes and restart glusterd again
23:04 a2 there might be a stale glusterd running somewhere
23:05 FilipeMaia I did /etc/init.d/glusterd restart
23:05 FilipeMaia Is there something else to do?
23:05 simpleirc1 joined #gluster
23:05 FilipeMaia looks like they don't really stop
23:06 FilipeMaia I'll do a killall glusterfs
23:07 FilipeMaia Still after restarting some nodes think there's a volume and others don't
23:07 FilipeMaia How can I clear this data?
23:08 a2 FilipeMaia, gluster volume sync
23:08 FilipeMaia what hostname?
23:08 FilipeMaia ah volname
23:08 a2 run it on the node which sees the volume
23:08 a2 and give hostname of a node which does not see the volume
23:09 FilipeMaia Looks hanged
23:09 hagarth joined #gluster
23:10 FilipeMaia Looks like the volume only disapeared from the node where I issued the volume delete command
23:11 FilipeMaia "gluster volume sync login" just seems to block
23:11 FilipeMaia It returned!
23:12 FilipeMaia but now gluster volume info is the one stuck
23:13 FilipeMaia after restarting glusterd  the volume is still there
23:14 primusinterpares joined #gluster
23:16 FilipeMaia There's this funny situation now http://paste.debian.net/10266/
23:16 glusterbot Title: debian Pastezone (at paste.debian.net)
23:16 FilipeMaia I can't stop the volume because it doesn't exist and I can't delete it because it's not stopped
23:18 FilipeMaia Is there a way to completely reset all glusterfs stuff?
23:23 FilipeMaia Now i'm also getting a "Host not connected" error
23:23 FilipeMaia But when running gluster peer status
23:23 FilipeMaia I get
23:23 FilipeMaia Hostname: a012
23:23 FilipeMaia Uuid: de6d52fc-2225-4410-b545-be9dacd77865
23:23 FilipeMaia State: Peer Rejected (Connected)
23:23 FilipeMaia For example
23:26 Hchl joined #gluster
23:28 FilipeMaia I deleted /var/lib/glusterd
23:29 FilipeMaia deleted the brick directories
23:29 FilipeMaia and that seemed to have reset things
23:30 jbrooks joined #gluster
23:33 dbruhn joined #gluster
23:34 rcoup I've ended up with a bunch of split-brain directories... most of the docs/help I've found is on fixing files (eg. http://joejulian.name/blog/fixing​-split-brain-with-glusterfs-33/)
23:34 glusterbot <http://goo.gl/FPFUX> (at joejulian.name)
23:36 rcoup they all have the same number of files
23:36 rcoup in fact, some are empty altogether
23:36 cakes joined #gluster
23:37 rcoup all* the two nodes
23:41 harold[MTV] joined #gluster
23:53 realdannys1 joined #gluster
23:54 rcoup I guess my question is - if I remove the .glusterfs attrs will it magically re-heal?
23:54 rcoup Or do I need to remove all the files? Or somethign else?
23:56 realdannys1 another push again to see if anyone can help - http://serverfault.com/questions/515507/c​ant-mount-glusterfs-volume-on-ec2-client
23:56 glusterbot <http://goo.gl/et1xL> (at serverfault.com)
23:59 JoeJulian rcoup: Yeah, I think that's my next blog...
23:59 JoeJulian rcoup: looks like you just have to zero out the trusted.afr.* ,,(extended attributes) on those directories on the bricks.
23:59 glusterbot rcoup: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary