Camelia, the Perl 6 bug

IRC log for #gluster, 2012-11-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:54 nightwalk joined #gluster
01:06 crashmag joined #gluster
01:15 kevein joined #gluster
01:25 stefanha joined #gluster
01:29 crashmag joined #gluster
01:31 khushildep joined #gluster
02:18 sunus joined #gluster
02:20 khushildep joined #gluster
02:30 sunus joined #gluster
02:47 bulde1 joined #gluster
02:57 esm_ joined #gluster
03:05 nightwalk joined #gluster
03:13 shylesh_ joined #gluster
03:13 shylesh joined #gluster
03:16 shylesh joined #gluster
03:21 ika2810 joined #gluster
03:22 bharata joined #gluster
03:32 saz joined #gluster
04:07 Shdwdrgn joined #gluster
04:14 crashmag joined #gluster
04:27 vpshastry joined #gluster
04:41 mdarade1 joined #gluster
04:46 khushildep joined #gluster
05:10 nightwalk joined #gluster
05:21 harshpb_ joined #gluster
05:24 faizan joined #gluster
05:34 Bullardo joined #gluster
05:43 mdarade1 left #gluster
05:48 sripathi joined #gluster
05:50 raghu joined #gluster
06:29 mohankumar joined #gluster
06:30 ankit9 joined #gluster
06:32 bulde__ joined #gluster
06:32 shireesh joined #gluster
06:40 vpshastry joined #gluster
06:53 Bullardo joined #gluster
06:54 blendedbychris joined #gluster
06:54 blendedbychris joined #gluster
06:58 ngoswami joined #gluster
07:04 harshpb_ joined #gluster
07:06 rodlabs joined #gluster
07:15 bala joined #gluster
07:24 ramkrsna joined #gluster
07:24 ramkrsna joined #gluster
07:27 rgustafs joined #gluster
07:28 TSM2 joined #gluster
07:35 ekuric joined #gluster
07:42 guigui3 joined #gluster
07:44 Nr18 joined #gluster
08:05 Azrael808 joined #gluster
08:12 hagarth joined #gluster
08:15 harshpb_ joined #gluster
08:15 maksim_ joined #gluster
08:16 ctria joined #gluster
08:17 TSM2 joined #gluster
08:17 maksim_ hi, i was thinking of serving my site images using a glusterfs so that the same images folder is shared among all my servers, is gluster fast enough to serve static files ?
08:19 guigui3 joined #gluster
08:22 harshpb_ joined #gluster
08:23 harshpb_ joined #gluster
08:24 abyss^ maksim_: gluster is fast such as cluster file system... But if you serve static files you should avoid to use any file system and retrieved directly from files from there... Use cache server and use memory - never file system. (I believe you understand my english ;))
08:25 abyss^ s/retrieved directly from files/retrieved directly files/
08:25 glusterbot What abyss^ meant to say was: maksim_: gluster is fast such as cluster file system... But if you serve static files you should avoid to use any file system and retrieved directly files from there... Use cache server and use memory - never file system. (I believe you understand my english ;))
08:27 TSM2 joined #gluster
08:27 ankit9 joined #gluster
08:37 gmcwhistler joined #gluster
08:42 vpshastry joined #gluster
08:52 Triade joined #gluster
08:55 puebele joined #gluster
08:58 andreask joined #gluster
08:59 Nr18 joined #gluster
09:25 puebele joined #gluster
09:48 tryggvil joined #gluster
09:53 sshaaf joined #gluster
10:03 mdarade1 joined #gluster
10:05 ankit9 joined #gluster
10:20 twx_ joined #gluster
10:23 twx_ hey guys, I have bricks in like, /data/brickA & /data/brickB - brickA is active in a volume already, but brick B i want to reuse for a new vol.. which means I'm running into the blabla or a prefix of it is already part of a volume-issue
10:23 glusterbot twx_: To clear that error, follow the instructions at http://goo.gl/YUzrh
10:23 twx_ yes, that doesn't work for me
10:24 twx_ getfattr -m .  -d -e hex shows no attributes for the directory whatsoever
10:24 twx_ trying setfattr -x also gives me error "no such attribute"
10:32 ctria joined #gluster
10:46 puebele left #gluster
10:54 vpshastry joined #gluster
11:02 layer3 joined #gluster
11:06 manik joined #gluster
11:17 harshpb joined #gluster
11:27 pkoro joined #gluster
11:38 puebele2 joined #gluster
11:44 mary_ joined #gluster
11:45 mary_ Hi, I remember there was a yum repo for gluster but I forgot it, can anybody help please ?
11:48 harshpb joined #gluster
11:56 manik joined #gluster
11:57 nightwalk joined #gluster
11:58 samppah @yum repo
11:58 glusterbot samppah: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
11:58 samppah manik: ^^
11:58 samppah mary :)
12:05 mary_ samppah: thx ^____^
12:06 twx_ noone ? :)
12:07 vpshastry joined #gluster
12:08 manik joined #gluster
12:10 dobber joined #gluster
12:40 puebele1 joined #gluster
12:42 harshpb joined #gluster
12:47 nightwalk joined #gluster
12:48 sunus joined #gluster
12:53 puebele joined #gluster
13:28 hagarth joined #gluster
13:35 mdarade1 left #gluster
13:37 ankit9 joined #gluster
13:39 esm_ joined #gluster
13:47 puebele joined #gluster
14:02 nueces joined #gluster
14:06 bennyturns joined #gluster
14:26 lh joined #gluster
14:26 lh joined #gluster
14:32 chaseh joined #gluster
14:37 bala joined #gluster
14:38 Dave2 Hey. I'm not sure if I'm completely using the wrong search terms here, but I can't seem to find a way to easily tell if a replicated volume is degraded (i.e. if there's a brick that can't be accessed); is there a way to check this? I'm aware of gluster peer status.
14:41 jdarcy gluster volume status
14:46 Dave2 jdarcy: I've tried that, but it doesn't seem to say that any of the bricks are missing, it just has a reduced number of lines of output
14:46 Dave2 is this to be expected?
14:52 balunasj joined #gluster
14:53 jdarcy Hm.  I thought it would show all the bricks, just with "Online" set to "N" for any that are down.  Might be a bug.
14:55 Bullardo joined #gluster
14:56 Dave2 That's what I was hoping to get; this is with gluster 3.3.1 on CentOS 6.3, in case that rings any alarm bells.
15:03 Bullardo joined #gluster
15:03 stopbit joined #gluster
15:05 rodlabs joined #gluster
15:05 jdarcy Seems rather non-intuitive and non-useful the way it is.  Would you mind filing a bug for that at https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS for us?
15:05 glusterbot <http://goo.gl/UUuCq> (at bugzilla.redhat.com)
15:08 twx_ jdarcy: sounds like u have someinvolvement in gluster, mind looking a few lines up at my question ? :)
15:08 sunus joined #gluster
15:14 bala joined #gluster
15:17 wushudoin joined #gluster
15:22 jdarcy twx_: You need to clean up some of the GlusterFS metadata before you can use a brick for a different volume.
15:22 jdarcy http://joejulian.name/blog/glusterfs-path-or​-a-prefix-of-it-is-already-part-of-a-volume/
15:22 glusterbot <http://goo.gl/YUzrh> (at joejulian.name)
15:23 * jdarcy giggles at "some involvement" BTW
15:24 pdurbin :)
15:24 twx_ jdarcy: heh, looked at that already, didn't help me much :)
15:26 twx_ also, I guess; s/some/much/ ?
15:27 pdurbin twx_: jdarcy is one of the core devs for gluster
15:28 twx_ I see
15:29 twx_ should be a good thing if I'm having trouble then, righT?:)
15:31 jdarcy Not necessarily.  Sometimes my bottom-up perspective isn't very useful for people looking in from the top.
15:31 jdarcy twx_: So did you try removing those xattrs and the .glusterfs directory?
15:31 twx_ yes, according to the link
15:32 davdunc joined #gluster
15:32 davdunc joined #gluster
15:32 twx_ running:  getfattr -m - -d -e hex /data/tank gives me nothing
15:32 twx_ also, if I even switch the brick name to something completely new, I get the same errormsg
15:33 twx_ could there be crazy xattrs in my root possibly?
15:38 jdarcy You seem to have an extra dash in that command, but that shouldn't matter.
15:40 jdarcy Is /data/tank a local-filesystem root, or is it a subdirectory of a local filesystem?
15:40 oneiroi joined #gluster
15:40 Dave2 jdarcy: will look into that, thanks</delayed>
15:41 twx_ a LVM LV is mounted under /data
15:42 twx_ i already have a brick under /data
15:42 twx_ which i dont want to mess about with
15:42 twx_ sure it's not production just playing around with gluster, but it'd be nice not to have to destroy that brick or or the fs it resides on
15:48 eurower joined #gluster
15:48 eurower Hi
15:48 glusterbot eurower: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
15:50 eurower I have a little problem with a replica 2 gluster volume. The volume is used so I can't stop it. The volume status indicates that one brick is offline and the other brick is online. How to make the first brick online ?
15:50 jdarcy Dave2: If it's a subdirectory, and it's empty, the easiest/safest thing to do is just remove it and mkdir a new one.
15:50 jdarcy eurower: gluster volume start $whatever force
15:51 Dave2 jdarcy: I think you may have mis-tabbed there
15:51 jdarcy Dave2: Ah, yes I did.
15:51 Dave2 (Incidentally, I've not got a registration e-mail from the Red Hat bugzilla yet - are you aware of any MTA issues with it?)
15:52 jdarcy eurower: That should do it, but it should have started automatically when glusterd started on that server.  If it failed then it might fail again for the same reason.
15:52 jdarcy Dave2: Not aware of any. OTOH, a lot of our infrastructure is sadly overloaded, so that kind of thing doesn't surprise me any more.
15:53 Dave2 I'll just be patient the
15:53 Dave2 n
15:53 eurower jdarcy : thanks a lot. The force option has changed offline to online, and synchronisation is done :)
15:53 noob2 joined #gluster
15:54 noob2 hey guys i'm seeing lack of quorum messages in my gluster logs on one machine.  all of the gluster nodes say all peers are connected.  I'm a little confused what the problem is
15:55 jdarcy eurower: Excellent.  :)
15:55 noob2 here's what i see in the log http://fpaste.org/vf2j/
15:55 glusterbot Title: Viewing Paste #250855 (at fpaste.org)
15:56 jdarcy noob2: Hey cool, somebody's actually using that feature.  :)
15:56 noob2 lol
15:56 jdarcy noob2: Just because all of the servers are up doesn't necessarily mean all of the bricks are up, or that clients (by which I mean processes, not necessarily machines) can reach them.
15:57 jdarcy noob2: So, first, what does "gluster volume status $whatever" say about the brick status?  Are all bricks present and online?
15:57 noob2 yeah
15:57 noob2 from the server side it looks like the bricks are up
15:57 noob2 let me scroll up to see if the client disconnected from one of them
15:57 * jdarcy was just about to suggest that.
15:58 jdarcy Also, what version is this?  ISTRC a bug where someone changed the information that quorum uses, it's fixed now but there might have been a version where it was broken.
16:01 ctria joined #gluster
16:01 noob2 jdarcy: looks like the gluster client disconnected a few days ago and i didn't notice
16:01 jdarcy Seems to be fixed in 3.3 and 3.3.1 though.
16:01 jdarcy noob2: And it hasn't reconnected?
16:02 noob2 checking
16:02 noob2 my client is this: glusterfs-fuse-3.3.0-8.el5 glusterfs-3.3.0-8.el5
16:02 puebele joined #gluster
16:02 noob2 my server is this: glusterfs-3.3.0-6.el6.x86_64 glusterfs-fuse-3.3.0-6.el6.x86_64 glusterfs-server-3.3.0-6.el6.x86_64
16:02 chouchins joined #gluster
16:03 noob2 lemme paste this.  it looks like it tried to fix it
16:03 noob2 http://fpaste.org/xwvc/
16:03 glusterbot Title: Viewing Paste #250861 (at fpaste.org)
16:04 noob2 jdarcy: i have an even number of machines so does quroum even matter?
16:04 noob2 i was wondernig about that the other day :)
16:05 noob2 i see the problem in the split brain logs
16:06 noob2 node 3 and 4 have differing versions and set the filesystem read only until i can repair it
16:06 noob2 which is good :)
16:07 puebele1 joined #gluster
16:11 daMaestro joined #gluster
16:12 jdarcy noob2: Congratulations on avoiding lots of split-brain.
16:12 eurower Humm. On brick A and B of replica 2 : resolv.conf is readable. But once mount the volume a third server, ls give : ??????????  ? ?    ?          ?            ? resolv.conf
16:13 jdarcy noob2: Quorum *currently* is only calculated within a replica set, so it still applies but can be a bit draconian.  With replica=2 that means you can get quorum failures because of a single failed brick, perserving the integrity of the data at the (high) cost of denying write-availability.
16:15 jdarcy noob2: In the near future, quorum will be counted either across all servers (not bricks) for the volume (not replica set), or even for the entire cluster.
16:15 jdarcy noob2: We might also add automatic assignment of observers/arbiters/tiebreakers whenever N=2, or promote to the next higher level of set/volume/cluster in those cases.
16:16 noob2 excellent :)
16:16 noob2 ok i removed the suspect file and did a stat on the client
16:16 noob2 it looks like the replicate failed
16:16 * jdarcy should write a blog post about this.  It's hard to describe in a few lines.
16:16 eurower Ok, it's a case of Split Brain. I said nothing ^^
16:16 noob2 http://fpaste.org/xN1B/
16:16 glusterbot Title: Viewing Paste #250875 (at fpaste.org)
16:16 jdarcy Does anyone know what happened to eurower?  He hasn't said anything for a while.  ;)
16:17 noob2 haha
16:17 elyograg jdarcy: I like that.  I'll be doing replica 2, so I had no plans to turn quorum on. It'll be nice to have quorum capability without a single failure making my volume read only.
16:17 eurower ohoh
16:17 noob2 yeah i have replica=2 on also
16:18 jdarcy Most people do.  The current quora implementation (and I'm the guy who wrote it) was a quick hack just to have *something* in place while we work on the real thing.
16:18 jdarcy s/quora/quorum/
16:18 glusterbot What jdarcy meant to say was: Most people do.  The current quorum implementation (and I'm the guy who wrote it) was a quick hack just to have *something* in place while we work on the real thing.
16:20 noob2 jdarcy: it looks like quorum is preventing my self heal
16:23 noob2 any idea what i should remove if the split brain logs only give me a gfid instead of a file location?
16:25 jbrooks joined #gluster
16:26 jdarcy noob2: This is from "gluster volume heal $whatever info split-brain"?
16:27 noob2 correct
16:28 noob2 btw, i wrote a little python script to use echo * to list files in a directory.  it's very fast compared to ls :)
16:29 jdarcy Yep, because it only does a readdir but no stat/getxattr.
16:30 noob2 yeah which removes glusters ability to do self heal right?
16:30 jdarcy If all you're getting is a gfid, then the only reasonable way to find the actual file is "ls -i .../.glusterfs/..." and then "find -inum"
16:31 noob2 ok
16:31 noob2 yeah it's been complaining about this gfid for weeks now
16:31 jdarcy Self-heal occurs on lookup, which precedes stat etc. but isn't necessary after readdir.
16:32 noob2 so could you point me in the right direction for this particular gfid if i show you the brick location?
16:32 noob2 i'm confused how fo find it :-/ sorry
16:33 noob2 oh i think i see what you're saying
16:34 noob2 find the inode on the gfid listed and then find by inum to remove
16:35 jdarcy ls -i /export/sdl/.glusterfs/c9/16/c916​03d6-a2ce-483c-8ac1-118e484cef25
16:36 jdarcy 198 /export/sdl/.glusterfs/c9/16/c916​03d6-a2ce-483c-8ac1-118e484cef25
16:36 jdarcy find /export/sdl -inum 198
16:36 jdarcy /export/sdl/.glusterfs/c9/16/c916​03d6-a2ce-483c-8ac1-118e484cef25
16:36 jdarcy /export/sdl/a_file
16:36 jdarcy Get the idea?
16:36 noob2 yup :)
16:36 noob2 thanks
16:36 noob2 lemme do that now
16:38 semiosis :O
16:39 jdarcy semiosis: Man, you gotta lay off those donuts.
16:39 JoeJulian Mmmm... donuts....
16:41 pdurbin JoeJulian: oh, you are here. gonna go bug you in #puppet-openstack
16:41 JoeJulian hehe
16:42 pdurbin your cover is blown
16:43 ika2810 joined #gluster
16:44 noob2 does write speed to the gluster cause split brains?  i have syslog writing to a gluster share and it seems to split brain often
16:44 semiosis ,,(split brain)
16:44 glusterbot I do not know about 'split brain', but I do know about these similar topics: 'split-brain'
16:45 semiosis meh i need to get consistent with my use of hyphens
16:45 semiosis ,,(split-brain)
16:45 glusterbot (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
16:45 noob2 haha
16:45 semiosis noob2: see #1, you're probably doing one of those anti-patterns that cause split brains
16:45 noob2 ok
16:47 noob2 i think i'm doing the 'later cause'
16:47 noob2 with syslog writing to a giant file
16:47 noob2 if a brick drops out it split brains
16:47 eurower hum, just to finish my test ^^. I have a replica 2 colume on server A and serveur B. On the gluster mount server C, I do a find /mnt/glustertest. On A and B I have a 23% CPU used. It is normal to consume so much CPU for fs ?
16:48 semiosis taking a break from glusterfs maintenance to do some web development in webstorm... you could say i traded split-brains for jetbrains
16:49 semiosis eurower: depends on the cpu i guess
16:50 eurower AMD FX(tm)-6200 Six-Core Processor
16:50 elyograg doesn't find do a stat on every file/dir, triggering the self-heal check and generally killing performance?
16:51 semiosis elyograg: yes, metadata operations are generally more expensive than data operations
16:51 semiosis enumerating all files in a volume is metadata intensive
16:51 semiosis so is stating them
16:53 Azrael808 joined #gluster
16:53 elyograg would it be possible to to write a glusterfs friendly version of find that works differently for better performance?
16:54 m0zes mlocate?
16:54 semiosis elyograg: i usually run find on my bricks if i have to search the whole fs
16:55 semiosis elyograg: though that's rare, and wouldn't work for "users" who need to find stuff
16:56 elyograg semiosis: that would be good for small setups.  my initial production rollout is going to have two servers, each with 22 bricks, replica 2.  that will quickly escalate to 4-6 servers.
16:57 elyograg 4TB drives, each divided into two bricks.  I would have done one brick per drive, but we have this huge pile of 2TB disks in our SAN hardware that the bean counters want to keep using once we decommission the SAN hardware.
17:01 cbehm elyograg: we specifically tell our users not to do an ls in some particularly poorly designed directories because of that. like semiosis, if i need to find something i do it via a brick
17:02 cbehm you could provide your users a tool for your client servers that did a find via key-based ssh back to one of the storage servers
17:03 noob2 semiosis: so is it safe to assume using gluster for syslogging is a bad idea?
17:03 noob2 unless your roll often
17:04 elyograg my main concern is with processes that determine disk usage for sections of the archive. someone in this channel told me that turning on quotas might give us up-to-the-moment usage numbers.
17:05 ctria joined #gluster
17:05 cbehm noob2: do you often have network or server failures? the split brain article seems to say that constant writes wouldn't be an issue unless you were (and didn't have some sort of quorum enforcement)
17:05 noob2 it's pretty rare that i have a network outage
17:06 noob2 it's usually one of the nics goes down for a second.  the thing is i have them 802.3ad aggregated
17:06 noob2 so i wouldn't think that is a problem
17:06 noob2 i set my lacp_rate to fast also to minimize downtime on the interface
17:07 cbehm noob2: we run a replica=3 setup with frequent little writes so i don't have direct experience with something like your syslog setup
17:07 jdarcy elyograg: One of the things we've talked about, which I might do in my spare time, is a "gfs_find" or "gfs_grep" that kicks off the appropriate commands locally on each brick (e.g. via a magic getxattr) and then aggregates the results into a virtual directory.
17:08 seanh-ansca joined #gluster
17:09 cbehm noob2: if the nic goes down, even briefly, and have a constant stream of data (and replica=2) that seems like it'd be all you'd need to get yourself split-brain
17:09 jdarcy elyograg: You could actually do something similar already, by running some sort of inotify-based daemon on the servers.
17:12 JoeJulian There could, theoretically, be a translator that would look at what application is running (since we have the pid) and translate that into a set of commands that could actually be managed on the servers. Would probably be a pain to maintain though. But say you "cp foo bar", the translator would get the first op, check the pid, see that it's a cp check that pid's environment for the pwd and run the cp directly on (or between) the servers. same c
17:12 JoeJulian ould be done for find and ls...
17:13 andreask joined #gluster
17:13 JoeJulian That would probably be a really good use case for glupy
17:13 jdarcy JoeJulian: Yup.
17:16 semiosis noob2: imnsho using anything other than logstash for logging is a bad idea :D
17:16 noob2 logstash?
17:16 semiosis noob2: though i'm sure there's a "safe" way to do syslogging on glusterfs, i just never really though much about it
17:16 semiosis @lucky logstash
17:17 glusterbot semiosis: http://logstash.net/
17:17 noob2 i'd say on the whole gluster has been really good for logs
17:17 noob2 the mgmt here is nervous about using gluster for other things until i can prove it's solid
17:20 noob2 although the oracle folks are starting to think about backing up to the gluster
17:22 cbehm noob2: even though there's a performance impact for additional replica levels, personally i don't think i'd do less than replica=3 (at least until there's something better for quorums and replica=2)
17:23 cbehm i don't want to deal with split-brain scenarios if i can help it :)
17:35 Mo__ joined #gluster
17:36 khushildep joined #gluster
17:40 Bullardo joined #gluster
18:06 mohankumar joined #gluster
18:10 zaitcev joined #gluster
18:17 rodlabs joined #gluster
18:18 noob2 cbehm: it seemed like the performance impact was significant when i tried replica=3
18:23 Bullardo_ joined #gluster
18:28 hagarth joined #gluster
18:29 nightwalk joined #gluster
18:42 elyograg i'm having lots of trouble getting gluster-swift working with s3.  cyberduck connects, but cannot do anything with directories in the bucket.  when i hack cyberduck's config file, i can get it to work with the swift api.  every time i find something that might be a potential avenue, keystone keeps coming up ... but i don't know anything about keystone, and the gluster-swift howto is using tempauth.
18:46 elyograg what we want is to use UFO, have multi-level directories, and have a consistent object API that we can connect to from various programming languages, primarily java.  S3 isn't a strict requirement, but it's something the devs keep talking about.
18:47 elyograg right this second I am looking at jclouds, but it seems to want keystone.
18:47 kkeithley keystone auth is in the works, but no promises for when.
18:49 elyograg if you can point me at the right doc sections so i can figure out how to do it myself, or give me some contact info for someone who has actually done it, it would be appreciated.
18:51 kkeithley Dunno. Off the top of my head I'd suggest the OpenStack-Swift docs. IIRC JoeJulian has used keystone for something.
18:51 elyograg I can promise that if I can figure out how to go about it, especially if I can figure out how to do it with a newer swift version, I'll update wikis or provide documentation additions so it'll help everyone.
18:52 kkeithley swift-1.7.4 is in the works too.
18:55 JoeJulian I installed openstack on centos6 using gluster's ufo in place of stock swift back in early July using bodepd's puppetlabs-openstack module. That installed essex and configured everything to use keystone so I really don't know how it was done specifically but you should be able to use puppet to do it.
18:55 JoeJulian The key to using keystone with ufo is that you have to use the keystone id for the volume name.
18:56 elyograg so it's similar to the relationship between the account & volume name in tempauth?
18:56 JoeJulian right
18:56 ika2810 left #gluster
18:59 elyograg ok, i'm installing the python keystone packages.  i guess i have some more googling to do.
19:02 hagarth joined #gluster
19:03 eurower joined #gluster
19:14 nueces joined #gluster
19:17 tc00per No office hours today?
19:23 wN joined #gluster
19:25 tryggvil_ joined #gluster
19:28 theron_ joined #gluster
19:29 JoeJulian No, johnmark and Technicool are in Barcelona. I was going to try to do it but I had to come in to the office.
19:29 tc00per Too bad... :(
19:30 tc00per ...we're not also in Barcelona.
19:30 JoeJulian Yesterday was just completely effed up for me, so I had to take my normal work-from-home day and come in to Seattle.
19:31 tc00per Bummer... how far is that for you? Ferry ride?
19:32 y4m4 joined #gluster
19:33 ctria joined #gluster
19:38 m0zes anyone here going to be at SC12
19:38 m0zes ?
19:38 JoeJulian No ferry ride. I take the train in. It's about a half hour total trip time.
19:40 elyograg JoeJulian: that's about what my daily train commute is from door to door, with a bus or shuttle at the work end.
19:41 elyograg http://www.rideuta.com/mc/?page=uta-home-trax
19:41 glusterbot Title: Utah Transit Authority (at www.rideuta.com)
19:41 davdunc joined #gluster
19:41 davdunc joined #gluster
19:43 JoeJulian Mine's a heavy rail with no stops in between. https://plus.google.com/photos/113457525​690313682370/albums/5629867161242860529
19:43 glusterbot <http://goo.gl/eMmr8> (at plus.google.com)
19:46 nick5 joined #gluster
19:49 elyograg bleh.  keystone uses mysql.  yet another service that I have to make redundant when I go live.
19:50 nick5 joined #gluster
19:52 elyograg or, to make things easier on myself later, make it redundant on my testbed.
19:52 berend joined #gluster
19:58 jack joined #gluster
20:02 nightwalk joined #gluster
20:05 tryggvil joined #gluster
20:28 sjoeboo_ joined #gluster
20:38 radev__ joined #gluster
20:40 ascii0x3F joined #gluster
20:40 ascii0x3F :0
20:43 radev83 hi, I'm new to gluster, trying to optimize it for small file read heavy setup, but it seems all docs are outdated for 3.3 version
20:44 radev83 I mean via gluster command you cannot set a lot of params
20:44 radev83 or change fuse translators
20:44 radev83 but editing volfiles seems to be discouraged now?
20:45 elyograg JoeJulian: I am to the point where I have a master-master replication setup of mysql and keystone has populated the mysql database.  now i'm going to try and add users/stuff to the db, and I need some clarification of your earlier statement about the keystone id, which has to the the same as the volume name.  keystone seems to have tenants, users, roles, tokens, and endpoints.  which of those did you mean by id?
20:50 semiosis radev83: correct, manually editing volfiles is not recommended, not supported, not wise, etc
20:50 radev joined #gluster
20:51 semiosis radev: correct, manually editing volfiles is not recommended, not supported, not wise, etc
20:51 radev_ joined #gluster
20:51 elyograg semiosis: third time's the charm? :)
20:52 semiosis fool me once...
20:53 semiosis radev_: correct, manually editing volfiles is not recommended, not supported, not wise, etc
20:53 radev_ My code is doing a lot of stats, is there any way to improve performance of that? I saw advise to use NFS client, but that kinda defies replication failover benefit
20:54 semiosis if your code is ,,(php) see below
20:54 glusterbot php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details.
20:55 radev_ my code is python
20:55 radev_ and it checks template last modified date to check if it needs to reload
20:56 semiosis that will cause a lot of stats
20:56 radev_ everything works with small number of templates
20:57 radev_ but when I change my load test to cover as much as possible - glusterfs hits 100% cpu
20:58 JoeJulian Is this django?
20:58 radev_ also, checking for file existence on non existent file - will this hit performance?
20:58 radev_ No, custom application (werkzeug, jinja2)
20:59 JoeJulian Yes, misses are expensive...
20:59 radev_ hmm, code was written with regular local fs in mind
20:59 JoeJulian Haven't I blogged that yet?
21:01 radev_ Can glusterfs fuse client utilize more than one core?
21:01 semiosis is your network really that fast?
21:02 semiosis are your disks?
21:02 semiosis cpu is rarely the bottleneck
21:02 radev_ https://s3.amazonaws.com/uploads.hipchat.​com/3897/11377/tvdbljga6u6kt0d/upload.png
21:02 glusterbot <http://goo.gl/Qy8hy> (at s3.amazonaws.com)
21:03 semiosis that's odd
21:03 semiosis what version of glusterfs?  also could you please ,,(pasteinfo)
21:03 glusterbot Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:05 elyograg you can have high cpu usage from bottlenecks other than the CPU.  while looking at that top output, press the number 1 so you can see all your CPUs.  One of them will probably show a very high %wa (iowait) percentage.
21:05 radev_ http://fpaste.org/2NZN/
21:05 glusterbot Title: Viewing gluster info by radev (at fpaste.org)
21:06 radev_ glusterfs 3.3.1 built on Oct 11 2012 21:49:36
21:08 semiosis not sure but i suspect maybe those diagnostic options you have enabled add some cpu overhead
21:09 radev_ http://fpaste.org/Orh8/
21:09 glusterbot Title: Viewing top output by radev (at fpaste.org)
21:09 radev_ This is per cpu top
21:09 radev_ no iowait
21:12 radev_ turned off profile
21:12 semiosis radev_: just a thought, but maybe you could have some global state in your app process that manages checking the file timestamp only once per second
21:12 radev_ same problem with CPU usage
21:12 semiosis instead of once per request, if that's what you're doing
21:13 radev_ Well, I had idea about storing timestamps in redis cache
21:13 radev_ instead of fs
21:13 elyograg just a quick glance at that, it appears to be a very busy box in general ... or is it only looking that way because it's all waiting on glusterfs?
21:13 semiosis another common strategy is to cache until invalidated, and make some kind of event fire to invalidate the cache of an object when it's updated in the backend/origin fs
21:14 radev_ we don't have a lot of servers, to this is my host for multiple VMs
21:15 radev_ webserver (has gluster client), one gluster node from pair, DB, redis
21:15 radev_ of course it's loaded under stress testing
21:16 elyograg JoeJulian: did you by chance see my additional question about keystone?  i've reached the point where I need to add all the stuff specific to my setup, but I don't see anything called an ID.  What does keystone call it?  My insinct is to think it's probably the tenant, but confirmation would be awesome.
21:16 jdarcy watching two people argue about how to interpret subsection 17.3(c) of some NIST recommendation (while ignoring much bigger security problems that don't happen to involve crypto fun) makes me want to pluck my own eyeballs out.
21:16 jdarcy Oops, wrong window.
21:18 radev_ but still, it's surprising that glusterfs is bottlenecked on CPU
21:19 radev_ and not on writes, but on reads
21:20 ctria joined #gluster
21:20 radev_ just in case - data from profile when I was stress testing
21:20 radev_ http://fpaste.org/awFi/
21:20 glusterbot Title: Viewing profile by radev (at fpaste.org)
21:20 elyograg radev_: i'm still new to gluster, but with everything i've read and everything that I've heard in this channel, I'm not surprised that a hevily utilized python app is killing one of your CPU cores like that.
21:21 elyograg can't type today.
21:22 elyograg JoeJulian: oh, I found the id you were talking about.  i really have to have a volume whose name is a 32 byte hex string?
21:22 elyograg http://www.gluster.org/2012/06/fedora-17-open​stack-essex-gluster-3-3-all-smushed-together/
21:22 glusterbot <http://goo.gl/qH8gb> (at www.gluster.org)
21:23 radev_ LOOKUP fop - what it's usually used for? stat?
21:26 jdarcy Lookup maps a name to an inode identifier (in our case a GFID) and is therefore the first thing we'll see for a file before stat/open/whatever.
21:30 radev_ hmm
21:30 radev_ volume top reports what period of time?
21:31 radev_ joined #gluster
21:32 blendedbychris joined #gluster
21:32 blendedbychris joined #gluster
21:36 m0zes_ joined #gluster
21:36 nightwalk joined #gluster
21:42 radev_ Simple cp -R /gfsshared/data/* /local/ command from gluster mount to local disk causes high CPU usage
21:43 radev_ is this normal?
21:43 jdarcy Probably high context-switch numbers too.
21:46 radev_ yes
21:46 andreask joined #gluster
21:47 radev_ is that because of fuse?
21:48 jdarcy Pretty much.
21:48 jdarcy In most situations the FUSE overhead is hard to notice, but directory listings are an exception.
21:49 jdarcy Since FUSE has no way to pass all the needed data (not just name but complete stat info and several xattrs) through in one call, we end up going back and forth more than we should.  My colleagues have submitted patches to improve that.
21:49 radev_ so I should use NFS client in that case?
21:50 jdarcy radev_: NFS client is a good thing to try if that's your use case.
21:50 radev_ but failover won't be automatic anymore, right?
21:53 jdarcy No, you'd have to use UCARP or something similar.
21:54 jdarcy NFS is a crappy protocol, but the NFS implementations have had the benefit of many developers over many years to optimize everything.
21:54 radev_ well, I thought that native gluster client would be faster
21:55 jdarcy It all depends on your workload.  For data yes, the the native protocol is likely to be faster.  For metadata, probably not.
21:56 radev_ so if I'll make only read calls - it'll be faster with native?
21:57 radev_ BTW, admin guide is really lacking in terms of describing caching - it's not clear if it's server or client side
21:57 jdarcy Reads can take advantage of caching, so if you have high locality then NFS might also be a better access protocol.  That's why we support it.
21:57 radev_ it seems you can't even set client properties with 'gluster volume set'
21:58 jdarcy Sure you can, it's just not clear which ones are server-side and which ones are client-side.  Most of the caching tunables are client-side.
21:58 radev_ and list is not full?
21:58 radev_ cause I saw some other params in sources
21:59 jdarcy Well, if you want to look at sources, you can see which ones are NODOC in .../xlators/mgmt/glusterd/src/glusterd-volgen.c - quite a few, actually.
22:00 radev_ ah
22:00 radev_ I was looking in individual xlators sources
22:00 jdarcy Yep, there are even more there, but of course you're pretty much on your own once you start hacking volfiles by hand.
22:01 radev_ can you change client chain with gluster CLI?
22:01 radev_ or server translator chain?
22:02 radev_ E.g. remove readahead?
22:02 twx_ jdarcy: did you have any clue of any other solution for my problem with reuse of a brick where the setfattr -x thingie didn't work ?
22:02 twx_ google couldn't find shit
22:03 jdarcy twx_: The only thing I can think of is that the old xattr values were cached, which might be cleared by restarting glusterd.
22:03 twx_ rebooted the box even
22:03 jdarcy twx_: Actually no, there is another possibility.  Sometimes if a management operation is interrupted, there are traces of the old volume config in /var/lib/glusterd/vols.  You might want to make sure there are no references left over there.
22:04 twx_ I'll check it out
22:04 jdarcy twx_: Make sure you purge any such remnants on all servers, or one server might "reinfect" the others.
22:07 radev_ performance.client-io-threads what would be effect of that switching on?
22:08 twx_ sorry jdarcy, can't find anything that indicates that it'd still be in use somehow
22:08 jdarcy radev_: Hard to tell.  Probably next to nothing for this case.
22:08 twx_ I've tried creating the volume several different times even with very different "paths" to the bricks btw. I.e. first /data/tank then /export/tank or even /export just
22:09 radev_ Ok, I'll experiment over the weekend
22:09 jdarcy twx_: Well, there are only two places that we store information about volumes - /var/lib/glusterd and the volumes themselves.  If you've removed all mention from both of those places and still get that error, I don't know what else to suggest.
22:09 radev_ Thanks
22:10 twx_ Alrite, I'll look around further then.
22:10 twx_ If I have a active brick
22:10 twx_ in lets say /data/blabla
22:10 jdarcy twx_: Are you sure that those attempts on different paths didn't leave xattrs on components of those paths?  Try looking all the way up to root.
22:10 twx_ could i safely clear xattrs on /data and / ?
22:10 jdarcy twx_: Nobody else is going to look at any of those xattrs, so yes, it should be safe.
22:11 nightwalk joined #gluster
22:11 twx_ under /data i find xattrs, however just with the volume name from another brick in /data
22:27 inodb joined #gluster
22:43 seanh-ansca joined #gluster
22:46 hattenator joined #gluster
22:52 stefanha joined #gluster
23:04 TSM2 joined #gluster
23:08 duerF joined #gluster
23:14 inodb left #gluster
23:43 t35t0r joined #gluster
23:54 andreask1 joined #gluster
23:58 wushudoin joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary