Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-04

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 khushildep_ joined #gluster
00:03 nage joined #gluster
00:03 nage joined #gluster
00:04 overclk joined #gluster
00:04 purpleidea sarkis: bbl tonight
00:08 JoeJulian purpleidea: Cool suggestion and result regarding cgroups
00:08 sarkis purpleidea: cool thanks!
00:08 jag3773 joined #gluster
00:19 ells_ joined #gluster
00:20 lluis joined #gluster
00:21 jporterfield joined #gluster
00:21 lluis Hello, Can someone briefly describe me the difference between "struct iovec" and "struct iobref" in the writev arguments?
00:21 ells joined #gluster
00:22 ells left #gluster
00:22 eryc joined #gluster
00:22 JoeJulian lluis: Sounds like a question that might get a better audience in #gluster-dev
00:24 dbruhn joined #gluster
00:25 ells_ joined #gluster
00:27 lluis JoeJulian: Thanks, I'll redirect it there.
00:39 sarkis joined #gluster
00:43 nixpanic joined #gluster
00:43 nixpanic joined #gluster
00:44 mtanner_ joined #gluster
00:49 gdubreui joined #gluster
00:56 mrfsl joined #gluster
00:56 mrfsl left #gluster
00:57 mrfsl joined #gluster
00:57 mrfsl is there a decent paper out there for performance tuning ?
01:00 gmcwhist_ joined #gluster
01:04 ira joined #gluster
01:20 mrfsl left #gluster
01:24 dbruhn mrfs1, not really. What are you trying to figure out?
01:25 hchiramm__ joined #gluster
01:32 tokik joined #gluster
01:33 vpshastry joined #gluster
01:36 gdubreui joined #gluster
01:37 gdubreui joined #gluster
01:52 jmarley joined #gluster
02:02 gdubreui joined #gluster
02:03 gdubreui joined #gluster
02:06 gdubreui joined #gluster
02:14 dbruhn joined #gluster
02:20 mtanner_ joined #gluster
02:24 gdubreui joined #gluster
02:26 mrfsl joined #gluster
02:27 mrfsl can someone enlighten me on the brick-replace changes in 3.4.2?
02:28 mrfsl I see now it is handled through self-heal - but the process is painfully slow
02:34 mrfsl also trying to do a: gluster volume heal <volname> info doesn't work (I am replacing a brick with millions of small files)
02:36 mrfsl i am tracking the progress by monitoring disk performance, network, and df -i (inodes) --- the inode count is increasing (albeit slowly) but network and disk utilization is practically zero
02:45 tjikkun joined #gluster
02:45 tjikkun joined #gluster
02:57 jporterfield joined #gluster
03:07 bharata-rao joined #gluster
03:15 jporterfield joined #gluster
03:23 drowe joined #gluster
03:23 shubhendu joined #gluster
03:24 drowe Hi all - I have a 2 node cluster that I haven't touched in quite some time (thus, my memory on repairing is failing me completely) - the IPs of one node changed, and they're not wanting to work together again
03:24 dbruhn drowe, what is the output of "gluster peer status"
03:24 drowe Many many months ago - I think someone here had me clear some settings files out of /etc/glusterd - but I can't for the life of me remember what I needed to remove to re-init the cluster and get the nodes back to talking
03:25 drowe From Node2 - Node1 is "Peer in Cluster (Disconnected)" - from  Node1, Node2 is "Peer in Cluster (Connected)"
03:26 dbruhn are you seeing IP addresses or hostnames?
03:26 drowe dbruhn: hostname - and /etc/hosts is up to date with the correct internal IPs
03:27 dbruhn have you restarted the glistered service on both servers?
03:28 drowe dbruhn - does the order in which it is restarted matter?
03:28 drowe ie, Node1 vs Node2 ?
03:28 dbruhn nope
03:28 dbruhn gluster is a masterless system
03:29 drowe okay - restarted glusterd on both nodes - peer status remains the same
03:31 dbruhn can you use fpaste and put the output of "gluster peer status", "gluster volume status", and "gluster volume info"
03:32 drowe well - hrmm - I touched a file to test, and it showed up - but Node1 is still reported as Disconnected
03:33 dbruhn it will if one is down
03:33 drowe why would a test file sync while peer status still reports a disconnect?
03:33 dbruhn what do you mean sync?
03:34 drowe I ran "touch drowe_was_here" on Node2 - then "ls" on Node1, and the file was present on both
03:34 dbruhn from the mount point or on the bricks?
03:34 drowe The mount point
03:35 dbruhn it would, because the client is connected to all available servers
03:36 drowe ah - you're right - I touched the file on Node2 in the mount point, and while it is showing up in the mount point on Node1 - it is NOT showing up in the brick on Node1
03:36 dbruhn yep
03:36 drowe Would that sound accurate for a Disconnected node?
03:36 drowe okay
03:37 LeonSandcastle joined #gluster
03:37 dbruhn can you use fpaste and give me the output of those commands I asked for before?
03:40 drowe dbruhn: http://fpaste.org/74163/39148522/  - gluster volume info
03:40 glusterbot Title: #74163 Fedora Project Pastebin (at fpaste.org)
03:41 drowe dbruhn: running 'gluster volume status' - both reported "unrecognized word: status (position 1)"
03:42 dbruhn what version of gluster are you running?
03:42 drowe 3.2.4
03:42 * drowe ducks
03:43 dbruhn hmm, there was a bunch of huge changes in 3.3, and to be honest I am mostly familiar with the in's and out's of that version
03:44 dbruhn what OS?
03:44 drowe I have to think it's something pretty basic - for node-prod-1a - it reports the Volume is 'stopped' - is it something stupid simple like unmounting the share, stopping gluster, starting, and remounting?  I can't think it'd be that
03:44 drowe Ubuntu
03:44 dbruhn are you running iptables?
03:49 drowe dbruhn - I think I owe you a beer
03:49 drowe or beverage of choice
03:49 dbruhn Glad it worked out.
03:49 drowe flushed iptables rules, and both report 'Connected'
03:49 drowe So, now should they sync up - anything that is out of sync?  Or how does that work?
03:50 dbruhn That I am not sure, I know 3.3 has a self heal daemon, but I am not sure about 3.2.x
03:50 dbruhn Is there any reason you haven't upgraded?
03:51 LeonSandcastle I'm lazy.. that's why we haven't upgraded.
03:52 drowe truthfully?  we were moving pretty fast, and learned enough to be dangerous - it was mostly a "set it and move on" scenario
03:52 dbruhn Fair enough, I would highly suggest getting to at least 3.3.2, if not up to the current 3.4 tree
03:52 LeonSandcastle Yeah.  It fell into the "...if it's not broke" bucket.
03:53 drowe dbruhn: with the replication - is it largely driven by accesses through the mount point?
03:53 drowe Not the brick?
03:53 dbruhn You should never be accessing the data from the brick directly ever, unless repairing something
03:53 dbruhn If you stat a file that is not on one brick, it should replicate it over to the other one
03:54 drowe dbruhn: bingo - that answers my question
03:55 drowe dbruhn - seriously - can I pay for coffee or something?
03:56 dbruhn Nah, just be part of the community. I've been in situations like yours, and people around here have helped me
04:05 kanagaraj joined #gluster
04:05 vpshastry joined #gluster
04:06 itisravi joined #gluster
04:18 ndarshan joined #gluster
04:19 kdhananjay joined #gluster
04:21 vpshastry left #gluster
04:27 davinder joined #gluster
04:35 purpleidea JoeJulian: thanks! I hope it's useful to you too.
04:36 purpleidea sarkis: sorry, afk before... did you figure out your replica problem?
04:37 JoeJulian purpleidea: I don't know that I'll use it, but it's sure a nice tool to have in my tool bag.
04:38 purpleidea JoeJulian: exactly, i feel the same way at the moment... i had a design for a gluster+vm's/openshift cluster which would have benefitted from cgroups, but at the moment, i'm not working on that project.
04:44 rjoseph joined #gluster
04:55 RameshN joined #gluster
04:57 vpshastry joined #gluster
04:58 bala joined #gluster
05:08 sputnik13 joined #gluster
05:08 sarkis purpleidea: no worries.. in the process of testing everything
05:08 sarkis our testing environment unfortunately has a ton of manual work :/
05:09 vkoppad joined #gluster
05:12 spandit joined #gluster
05:12 haomaiwa_ joined #gluster
05:13 purpleidea well, keep automating!
05:19 hagarth joined #gluster
05:21 rastar joined #gluster
05:23 prasanth joined #gluster
05:23 shylesh joined #gluster
05:23 nshaikh joined #gluster
05:26 saurabh joined #gluster
05:27 zapotah joined #gluster
05:27 zapotah joined #gluster
05:28 sarkis yea thats the goal
05:28 sarkis gonna kill 2 birds, getting the puppetmaster dev env figured out while im working through this
05:28 sarkis s/kill 2 birds/kill 2 birds with 1 stone/
05:29 glusterbot What sarkis meant to say was: gonna kill 2 birds with 1 stone, getting the puppetmaster dev env figured out while im working through this
05:29 mohankumar joined #gluster
05:32 dusmant joined #gluster
05:36 lalatenduM joined #gluster
05:42 RameshN joined #gluster
05:43 ppai joined #gluster
05:47 tokik joined #gluster
05:47 raghu joined #gluster
05:49 vimal joined #gluster
05:49 sarkis joined #gluster
05:50 _dist joined #gluster
05:52 mohankumar joined #gluster
05:56 benjamin_____ joined #gluster
06:00 vpshastry1 joined #gluster
06:02 davinder2 joined #gluster
06:04 shylesh joined #gluster
06:10 fidevo joined #gluster
06:13 jikz joined #gluster
06:17 Philambdo joined #gluster
06:18 fidevo hello guys, is manpages for gluster available for download anywhere in download.gluster.org?
06:23 aravindavk joined #gluster
06:23 harish joined #gluster
06:29 shapemaker joined #gluster
06:29 sarkis_ joined #gluster
06:30 Philambdo joined #gluster
06:30 mohankumar joined #gluster
06:31 _dist I'm having a bit of trouble with qemu live migration, libvirt keeps complaining it can't stop block jobs right at the end. The migration finishes but the guest at its' new location can't write to it's file until it's power cycled
06:31 haomaiw__ joined #gluster
06:31 _dist Are there any special setup steps I might have missed? (gluster is on xfs)
06:32 edong23_ joined #gluster
06:32 dbruhn_ joined #gluster
06:32 hflai joined #gluster
06:32 toki joined #gluster
06:33 social joined #gluster
06:33 tru_tru_ joined #gluster
06:34 purpleid1a joined #gluster
06:34 l0uis_ joined #gluster
06:34 mkzero_ joined #gluster
06:37 sac`away` joined #gluster
06:37 atrius` joined #gluster
06:37 jporterfield_ joined #gluster
06:44 mojorison joined #gluster
06:45 benjamin_____ joined #gluster
06:45 simon__ joined #gluster
06:45 tziOm joined #gluster
06:45 harish joined #gluster
06:46 ndarshan joined #gluster
06:46 rastar joined #gluster
06:46 fidevo joined #gluster
06:47 nshaikh joined #gluster
06:48 TvL2386 joined #gluster
06:48 hybrid512 joined #gluster
06:50 primusinterpares joined #gluster
06:51 aravindavk joined #gluster
07:02 bharata-rao joined #gluster
07:02 Slasheri joined #gluster
07:02 Slasheri joined #gluster
07:03 haomai___ joined #gluster
07:13 dusmant joined #gluster
07:21 mrfsl joined #gluster
07:27 jtux joined #gluster
07:27 mohankumar joined #gluster
07:32 asku joined #gluster
07:38 jporterfield joined #gluster
07:39 mohankumar joined #gluster
07:46 mohankumar joined #gluster
07:47 StarBeast joined #gluster
07:49 vpshastry1 joined #gluster
07:50 ekuric joined #gluster
07:53 pk1 joined #gluster
07:57 aravindavk joined #gluster
07:57 dusmant joined #gluster
07:58 ctria joined #gluster
08:16 VerboEse joined #gluster
08:17 VerboEse joined #gluster
08:20 pk1 joined #gluster
08:21 badone_ joined #gluster
08:39 zapotah joined #gluster
08:43 pk1 joined #gluster
08:44 ndevos @later tell _dist make sure you have checked these prerequisites: http://www.ovirt.org/Features/GlusterFS_​Storage_Domain#Important_Pre-requisites
08:44 glusterbot ndevos: The operation succeeded.
08:46 vpshastry1 joined #gluster
08:48 20WAA24RF joined #gluster
08:49 harish joined #gluster
08:59 calum_ joined #gluster
09:05 shylesh joined #gluster
09:07 harish joined #gluster
09:10 pk1 joined #gluster
09:11 andreask joined #gluster
09:12 eseyman joined #gluster
09:31 ninkotech joined #gluster
09:31 ninkotech_ joined #gluster
09:39 bala joined #gluster
09:43 surabhi joined #gluster
09:47 CheRi joined #gluster
09:49 badone joined #gluster
09:53 blook joined #gluster
09:54 meghanam joined #gluster
10:00 vpshastry2 joined #gluster
10:00 ells joined #gluster
10:03 jikz joined #gluster
10:05 pk1 left #gluster
10:07 hybrid512 joined #gluster
10:09 liquidat joined #gluster
10:12 vpshastry1 joined #gluster
10:15 andreask joined #gluster
10:16 atrius joined #gluster
10:24 jporterfield joined #gluster
10:26 rjoseph joined #gluster
10:32 qdk joined #gluster
10:44 jikz joined #gluster
10:49 ira joined #gluster
10:51 DV joined #gluster
10:54 matclayton joined #gluster
10:54 matclayton Hey guys, we're running 3.3.1 in a Replicate set, and just lost a brick (a large brick, 45TB RAID6 volume), any advice for recovering it?
10:55 ctria joined #gluster
10:55 matclayton The other replica has taken over and we're still online, we now just need to get the data replicated again
11:01 vpshastry1 joined #gluster
11:05 jporterfield joined #gluster
11:09 spandit joined #gluster
11:19 hybrid512 joined #gluster
11:21 spiekey joined #gluster
11:23 Philambdo joined #gluster
11:24 nshaikh joined #gluster
11:28 spiekey left #gluster
11:31 pk1 joined #gluster
11:39 hybrid512 joined #gluster
11:40 bala joined #gluster
11:48 shubhendu joined #gluster
11:49 P0w3r3d joined #gluster
11:53 ndarshan joined #gluster
11:54 dusmant joined #gluster
11:56 spandit joined #gluster
11:57 kkeithley1 joined #gluster
12:01 ctria joined #gluster
12:04 glusterbot New news from newglusterbugs: [Bug 1061069] Unable to start glusterd service because of missing dependencies <https://bugzilla.redhat.co​m/show_bug.cgi?id=1061069>
12:04 harish joined #gluster
12:07 zapotah joined #gluster
12:07 zapotah joined #gluster
12:10 andreask joined #gluster
12:16 DV joined #gluster
12:24 itisravi joined #gluster
12:26 pk1 left #gluster
12:32 twx joined #gluster
12:41 zapotah joined #gluster
12:41 zapotah joined #gluster
12:50 hagarth joined #gluster
12:51 rfortier2 joined #gluster
12:52 surabhi joined #gluster
13:00 bala1 joined #gluster
13:08 rcaskey semiosis, thanks for your referral to gluster volume set help yesterday. I had read it before but overlooked/didn't grok the significance of 'exactly half includign the first.'
13:09 davinder joined #gluster
13:11 Ark_explorys joined #gluster
13:12 rcaskey assuming it existed in an alternate universe where it was present from the very beginning, do you think auto would be a sane default value or are there good reasons where allowing split-brain by default makes sense?
13:13 benjamin_____ joined #gluster
13:16 ctria joined #gluster
13:19 shubhendu joined #gluster
13:22 matclayton joined #gluster
13:22 mohankumar joined #gluster
13:28 cloud joined #gluster
13:29 Guest49620 does glusterfs 3.3.2 support ipv6
13:29 Guest49620 ?
13:36 vimal Guest49620, http://www.gluster.org/pipermail/glu​ster-users/2013-November/037824.html
13:36 glusterbot Title: [Gluster-users] Hi all! Glusterfs Ipv6 support (at www.gluster.org)
13:41 mrfsl joined #gluster
13:42 mrfsl Good morning Oh Gods of Gluster! --- Here's a question for you. I have some brick-replaces to do; migrating replica bricks to a new server. Can I use dd to jump start the operation?
13:42 rcaskey Guest49620, btw use /nick YourNewNick and make yourself at home :)
13:42 purpleidea joined #gluster
13:43 rwheeler joined #gluster
13:44 sroy_ joined #gluster
13:44 rcaskey mrfsl, how big a brick we talking here?
13:45 mrfsl 4 TiB
13:46 rcaskey dunno, I'm a newb, but that doesn't seem all that big these days - I'd be tempted to add a new node to the cluster, remove the old one, and rebalance?
13:46 mrfsl have been doing that now for several several hours - and have moved very little data
13:47 rcaskey you got a gazillion small files?
13:47 mrfsl of course! Haha! So no one can help me right?
13:48 rcaskey I can't, someone else may be able to. I'm wondering if you would end up doing cleanup on attrs that ended up having similar runtime charactersitics
13:48 mrfsl I was wondering can I 1) dd the replica brick to the new destination 2) Brick Replace 3) Self Heal?
13:50 mrfsl Don't know - I have been scouring the webs for more information - any information. If anyone can help me out with insights into:
13:50 mrfsl 1) How the "new" brick-replace method in 3.4.2 works
13:50 mrfsl 2) How self-heal works
13:50 mrfsl 3) How to speed up #2
13:50 mrfsl 4) Alternatives to #2
13:50 mrfsl I would be eternally grateful! :D
13:52 mrfsl Is there a good Gluster "deep-dive" document that someone can recommend - I don't mind engineering my own solution
13:53 * rcaskey watches mrfsl's excesively long list of questions...and will blatently steal his answers
13:53 mrfsl haha! Excellent!
13:54 rcaskey but I was wondering the same thing... is there a 'more advanced concepts' sequel to gluster concepts
13:55 rcaskey as a new comer the appearance is that a lot of this sort of stuff was handled back in the day at the filesystem level and people mostly understood the book keeping that was going on in the background
13:55 mrfsl I want to be one of those people!
13:55 rcaskey and over time that is becoming less and less true because there are certain limitations inherit in that process that are being worked around
13:55 mrfsl How do I begin! ;D
13:55 jmarley joined #gluster
13:56 bennyturns joined #gluster
13:57 kkeithley_ jdarcy's http://cloudfs.org/index.php/blog/ goes into detail on a variety of glusterfs concepts. His talks at Red Hat Summit also go into some more detail.
13:57 rcaskey and that if we come back in 10 years time eventually none of that old backing will be there, we won't be able to manipulate those files, and we will be glad that the lower-level copy is just done on our behalf when feasible
13:58 * rcaskey has to run to a meeting but will read that presentation when he gets back
14:00 mrfsl I inherited this environment. I have 200 TiB of data between 3 nodes with 192 GiB of RAM on each node and 10 Gb network. I just want the self heal to move faster than 168.98 Kb/s. --- Thanks rcaskey for your sympathy.
14:03 matclayton Hey guys, we're running 3.3.1 and had a brick blow up (45TB RAID6) we were running replicated, and have rebuilt the RAID6 array and preparing to replicate it back. Any advice, this is a live cluster and reenabling the process on that brick nukes out production io
14:03 matclayton or at least nginx locks up
14:03 matclayton serving from it
14:05 DV joined #gluster
14:06 keytab joined #gluster
14:10 NuxRo matclayton: have you tried renicing the gluster process?
14:10 theron joined #gluster
14:10 NuxRo just an idea
14:10 matclayton renicing?
14:10 matclayton no we haven't took a while to figure that one out
14:11 theron joined #gluster
14:12 dbruhn joined #gluster
14:12 itisravi joined #gluster
14:14 theron joined #gluster
14:17 plarsen joined #gluster
14:18 purpleidea matclayton: we just talked about this on the ml the other day... cgroups is probably the tool you want. forget renicing.
14:18 matclayton would you recommend bumping to 3.4.2 before doing this repair?
14:19 purpleidea matclayton: i think (barring some exceptions) try to solve one problem at a time.
14:19 purpleidea eg: don't try and do an upgrade+repair as one big job.
14:20 matclayton purpleidea: thats our current thought
14:21 purpleidea good luck then!
14:24 matclayton is there a way to disable replicate and read, and manually do them?
14:24 matclayton *replicate-on-read
14:30 B21956 joined #gluster
14:30 rwheeler joined #gluster
14:31 gmcwhistler joined #gluster
14:31 bala joined #gluster
14:35 ctria joined #gluster
14:39 benjamin_____ joined #gluster
14:39 gmcwhistler joined #gluster
14:47 keytab joined #gluster
14:49 CheRi joined #gluster
14:56 failshell joined #gluster
14:59 dewey joined #gluster
15:01 dewey Question on brick replacement:  I currently have replica=2 with bricks of 36TB.  I'm only using 16T and I'm about to migrate to a new node.  I'd like smaller brick size for managability.  Can I run replace-brick on an existing 36T brick to a new 18T brick?  I know the data will fit, but will Gluster object or will it "do the right thing" and just consider the brick pair to be 18TB (i.e....
15:01 dewey ...wasting 18T of my other brick, which I'll fix later).  Thanks.
15:06 jobewan joined #gluster
15:06 rastar joined #gluster
15:07 RameshN joined #gluster
15:21 sroy joined #gluster
15:21 lalatenduM joined #gluster
15:24 pixelgremlins hey... I have 3 servers... files that get pushed to /var/www are put on server2:/var/export/www server3:/var/export/www -- vice versa... anything on any server in var/www gets mirrored... -- but say there's session data and someone is on server two.. files put into var/export/www doesn't get shared across nodes... could I symlink var/export/www to var/www what's the best way to handle dynamic nodes
15:25 dbruhn pixelgremlins, are you accessing the data in the bricks directly or through the mount point right now?
15:26 pixelgremlins my coding process is basically - write local. push to server via git. to server1:/var/www that goes to servers 2 and 3 @ /var/export/www,  etc... but there's sessions/cache that needs shared back across all servers..
15:26 JoeJulian pixelgremlins: the bricks are storage for GlusterFS, not for your application. You must mount the volume to make use of the features.
15:27 JoeJulian ... and, imho, session data should be in memcached or some equivalent.
15:27 jobewan joined #gluster
15:27 pixelgremlins not accessing it yet-- I'm trying to figure out - well there's also log files and such, -- using Laravel and there's a /storage folder that needs shared,
15:28 pixelgremlins I'm trying to figure out the best course of action- - I was using Csync + Lsyncd -- but that ran into hiccups and was unreliable
15:29 pixelgremlins This is basically for nodebalancing - where each node needs to be identical.
15:29 JoeJulian Why?
15:29 pixelgremlins Because I also don't want to have to run composer update on every server--so when I run it on server 1 it should also just push out to all other servers
15:30 JoeJulian So your goal is that all your servers have the same source code.
15:30 pixelgremlins exactly
15:30 pixelgremlins update on one-- and it updates all.
15:30 pixelgremlins regardless of which server is updated
15:30 JoeJulian so... you don't need to have them all identical. You just need to have them all access the same stored source.
15:31 pixelgremlins there's also file uploads that upload locally -though I will eventually be implementing a CDN, or central store for that.
15:31 JoeJulian Once volume with the source. All servers would run from that one volume.
15:31 pixelgremlins I want if someone uploads a file from server 3, for it to be accessible on server 1 and 2
15:31 pixelgremlins so I want it to be bidirectional
15:32 JoeJulian Right. That's why it's on the volume. All files anywhere on the volume are accessible to every client that mounts that volume.
15:32 JoeJulian GlusterFS isn't a synchronization service, it's a clustered filesystem.
15:32 rossi_ joined #gluster
15:33 pixelgremlins But it can do what I want? I'm a newb to devop stuff /sysadmin - trying to find the best options here.
15:34 JoeJulian Sure. Being a newb, I recommend you read ,,(php)
15:34 glusterbot (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://joejulian.name/blog/optimizi​ng-web-performance-with-glusterfs/ for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
15:34 glusterbot --fopen-keep-cache
15:35 glusterbot New news from newglusterbugs: [Bug 1061229] glfs_fini leaks threads <https://bugzilla.redhat.co​m/show_bug.cgi?id=1061229>
15:36 calum_ joined #gluster
15:40 bugs_ joined #gluster
15:51 _dist joined #gluster
15:56 daMaestro joined #gluster
16:08 harish joined #gluster
16:15 davinder joined #gluster
16:20 sarkis_ joined #gluster
16:22 zerick joined #gluster
16:22 matclayton Is it possible to rsync a replica set to recover it?
16:25 RameshN joined #gluster
16:27 sarkis_ purpleidea: so i think your common module should now be required, since you can't specify a volume as the gluster::volume class includes gluster::again and that includes common::again
16:27 sarkis_ rather than optional*
16:30 semiosis pixelgremlins: why not just use heroku or some other PaaS hosting?  why do you need to run your own HA PHP web servers?
16:30 purpleidea sarkis: woops! you're quite right... i had thought i had added a flag for that feature, but i guess i forgot. i'll try and patch it shortly.
16:31 purpleidea sarkis: care to email ping me a reminder?
16:31 sarkis you will be adding a flag to enable/disable?
16:31 sarkis sure
16:31 jag3773 joined #gluster
16:31 semiosis pixelgremlins: if you really need to do that, i'd suggest looking into a deployment system that will store the site files on the servers in local storage... glusterfs can serve php, and i've done it, but if you dont have to you might want to go a different route
16:31 purpleidea sarkis: yeah, all the optional features have a default and a flag to enable/disable. eg shorewall => true/false, vrrp => true/false, etc...
16:32 purpleidea i guess i forgot to add the exec again flag, but it was my intention. congratulations you've found the first (sort of) bug
16:32 pixelgremlins I'm using Linode VPS seemed more affordable.
16:32 sarkis ty ty :)
16:32 semiosis pixelgremlins: what i do now is store all the web code in gitlab & have a commit hook which runs composer, tars up the site, then uploads the tarball.  the web servers download & untar the archives & serve them
16:32 sarkis purpleidea: what was your email again? sorry forgot to log it
16:32 semiosis also store sessions in memcache or something, not files.
16:33 pixelgremlins what about user uploads?
16:34 _dist morning :) last night I was asking about an issue with live migration where I'm getting libvirt error "unable to stop block job". I'm not positive it's a gluster issue. Last night someone gave me this link when I was afk http://www.ovirt.org/Features/GlusterFS_​Storage_Domain#Important_Pre-requisites but I'm not running ovirt, or using libgfapi. Hoping someone else has seen this issue
16:34 pixelgremlins sessions and cache are in redis on a central server
16:34 _dist The migration works, but the disk is ro on the dest side until power cycle
16:35 pixelgremlins but some plugins also put data into the app/storage folder which may end up breaking if it's not synced.
16:36 semiosis pixelgremlins: ah i see
16:36 bala joined #gluster
16:37 semiosis pixelgremlins: well you may want to consider a hybrid approach, where you deploy code like i was saying, but keep the user data in a mounted storage volume
16:37 JoeJulian _dist: Which distro?
16:38 pixelgremlins does your commit hook send the file via SCP to all the servers?
16:39 JoeJulian Rackspace does their deploys using bittorrent.
16:39 semiosis pixelgremlins: the commit hook basically does this... 1. clone repo to tempdir, 2. run composer install --no-dev, 3. tar up the tempdir, 4. upload tarball to S3
16:39 _dist joejulian: My initial setup was ubuntu 13.10, on gluster 3.4.2 from semiosis ppa, on xfs, with libvirt 1.2.1 and qemu 1.7.0. However, I've tried stock ubuntu qemu+libvirt as well, and ext4 instead of xfs. I've tried the auth issue (even though I believe that's a libgfapi mount thing)
16:39 zaitcev joined #gluster
16:40 semiosis pixelgremlins: then the web servers have a script on a cron job that checks S3, compares local tarball md5s with the md5s in s3, and downloads & untars any that are new or modified into /var/www
16:40 mohankumar joined #gluster
16:40 semiosis pixelgremlins: we host many sites
16:40 _dist joejulian: also, I've done the port change per https://www.mail-archive.com/glust​er-users@gluster.org/msg14256.html
16:40 glusterbot Title: Re: [Gluster-users] gluster and kvm livemigration (at www.mail-archive.com)
16:40 semiosis this system has proven extremely effective both in deployment reliability and developer happiness
16:40 JoeJulian _dist: cool. Was just curious since that's been reported (so far) by only ubuntu users.
16:41 semiosis JoeJulian: istr somethign about facebook using BT also... might be confused tho
16:41 * JoeJulian isn't pointing fingers, just noticing...
16:42 semiosis pixelgremlins: oops, actually not a cron job, the script on the servers is run by puppet.  it also can modify the apache config, adding vhosts defs for the sites, but that's probably not relevant for you
16:43 _dist joejulian: didn't realize that, I'm comfortable with most distros, and have honestly had migration working in the past on ubuntu 13.10. I really like to get to the bottom of things like this rather than reinstall or switch. Can you think of anyway I can get "deeper" into what's going on? I suppose I could try live migration on regular NFS to narrow it down...
16:44 chirino joined #gluster
16:45 bala1 joined #gluster
16:46 JoeJulian _dist: I agree, that's what I like to do too. :D If I were diagnosing it, I'd use trace logs on the client/server to see what steps occur that might cause a deadlock. Failing to find anything there, wireshark and/or strace. File a bug report with your findings and a state dump (sigusr1) from the client and the brick(s) that hold that image.
16:46 semiosis JoeJulian: this problem is specific to ubuntu?
16:46 semiosis first i've heard of it
16:47 JoeJulian semiosis: so far, unless there's just nobody migrating vms in EL.
16:47 semiosis _dist: afaik the qemu in ubuntu doesnt support gluster natively... did you build it yourself?
16:48 JoeJulian semiosis: He's not using the api, just the fuse mount.
16:49 semiosis oh hmmm
16:49 semiosis turn off write-behind
16:49 matclayton joined #gluster
16:49 _dist semiosis: on the gluster volume, got it I'll try that right now
16:49 semiosis JoeJulian: what other perf xlators that are on by default could weaken consistency?
16:49 JoeJulian And that's been the case with 4 different people that have reported this issue (none with bug reports... grr...)
16:50 JoeJulian The work-around, so far, has been to use the api and get better performance along with it.
16:50 semiosis hey i'm all for that!
16:50 semiosis btw, ubuntu people are looking into making qemu/gluster part of the official ubuntu qemu packages
16:50 JoeJulian I'm thinking it's some sort of locking issue.
16:50 _dist joejulian: I will :) honestly the reason I'm not using the api is because I don't like ovirt (another topic) and no version of virt-manager (even 0.10) supports gluster pool management that I can find)
16:51 JoeJulian openstack
16:51 vpshastry joined #gluster
16:51 samppah _dist: afaik it's not currently even possible to create gluster pools with libvirt that would use libgfapi
16:52 _dist joejulian: I don't mind using virsh and editing xml, but I'm trying to provide something more elegent for the people I work with, yeah I know virt-manager isn't that elegent
16:52 _dist samppah: right but libvirt 1.2+ can use libgfapi as a storage pool, that I have got to workbut it's xml editing for the machines
16:52 JoeJulian Now I've got to check that out. I was sure I saw gluster support in libvirt/virt-manager...
16:53 JoeJulian hmm, nope... Maybe I was dreaming...
16:53 kkeithley_ yes, there's support for glusterfs in libvirt, starting with f19.
16:53 _dist joejulian: yeah libvirt only as far as I can tell, I didn't see any compile options in the new virt-maanger (but honestly without the new python.py options I'm not as quick as I was with ./configure)
16:55 _dist At home I'm running libvirt1.2.1 and qemu1.7.0 both custom compiled for libgfapi, the storage pool works fine but virt-manager gui has weird error messages if you try and add a glusterfs pool (type 10) through the gui
16:55 samppah btw, currently libgfapi is not working with ovirt either :/
16:56 JoeJulian theron: ^
16:56 JoeJulian Oh?
16:57 samppah libvirt doesn't support snapshots with libgfapi so it's currently disabled in ovirt
16:57 samppah https://bugzilla.redhat.com​/show_bug.cgi?id=1022961#c1
16:57 glusterbot Bug 1022961: urgent, urgent, ---, amureini, NEW , [BLOCKED] Running a VM from a gluster domain uses mount instead of gluster URI
17:00 diegows joined #gluster
17:02 vpshastry left #gluster
17:03 _dist tried write-behind disable, restarted volume still got the same issue. I'll be back in a few min just grabbing a sub but working through this issue, is what I'm doing today :)
17:04 semiosis _dist: please ,,(pasteinfo) when you have a chance
17:04 glusterbot _dist: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
17:05 purpleidea the bot is really getting good. wow it knows gluster commands
17:12 JonnyNomad joined #gluster
17:13 sarkis ya that is cool
17:13 sarkis it's gonna replace us! damn machines
17:14 theron joined #gluster
17:15 Fresleven joined #gluster
17:15 KyleG joined #gluster
17:15 KyleG joined #gluster
17:16 bala joined #gluster
17:17 matclayton Anyone seen self heals locking up nginx talking to glusterfs-native-client
17:17 theron thanks JoeJulian
17:19 JoeJulian matclayton: Not since 3.3.0
17:19 matclayton we're in 3.3.1
17:20 * glusterbot thinks about saving humanity from itself... maybe in August...
17:21 matclayton JoeJulian: any idea what ticket/bug was relevant?
17:22 rastar joined #gluster
17:26 LoudNoises joined #gluster
17:27 JoeJulian oldbug 2949
17:27 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=764681 is not accessible.
17:28 JoeJulian odd
17:28 hagarth JoeJulian: fixed
17:28 glusterbot New news from resolvedglusterbugs: [Bug 764681] self-heal hangs <https://bugzilla.redhat.com/show_bug.cgi?id=764681>
17:29 matclayton hmm ok, any memory if it was fixed for 3.3.1 or should we be upgrading?
17:29 JoeJulian Always worked for me in 3.3.1...
17:29 hagarth matclayton: fixed in 3.3.0 me thinks
17:30 Theory joined #gluster
17:30 JoeJulian The 3.3.0 bugs were for other things...
17:30 JoeJulian mostly management issues
17:31 matclayton hmm ok, essentially we lost a brick (RAID6 volume) and trying to recover it, as soon as we bring the brick back online the nginx layer talking to glusterfs via the native mount entirely locks up
17:31 matclayton this is large brick, 45TB
17:32 semiosis is there heal activity going on?
17:33 semiosis possible that there's so much healing the performance is just severely degraded?  that's easy to fix with background self heal count
17:33 Mo__ joined #gluster
17:33 matclayton semiosis: I can't get the log, it timesout
17:34 JoeJulian hagarth: What happens when the background self-heal queue reaches the self-heal count and one more file is opened? I've always assumed the read would have to wait for a self-heal slot to open.
17:34 JoeJulian Or for a foreground self-heal to complete...
17:35 hagarth JoeJulian: yes, once the self-heal-count is reached, operations will block till a slot is available.
17:36 matclayton So all we know is we've had it turned on for around 10minutes and its healed around 25GB
17:37 matclayton hagarth: Is there anyway to disable the clients from running the self-heal, so our web layer continues serving, and we touch all the files in the background to heal them, or let the shd do a pass?
17:37 JoeJulian @undocumented
17:37 glusterbot JoeJulian: I do not know about 'undocumented', but I do know about these similar topics: 'undocumented options'
17:37 JoeJulian @undocumented options
17:37 glusterbot JoeJulian: Undocumented options for 3.4: http://www.gluster.org/community/documentat​ion/index.php/Documenting_the_undocumented
17:38 Humble joined #gluster
17:39 JoeJulian Hmm, I thought I saw it there...
17:39 zerick joined #gluster
17:39 KyleG left #gluster
17:39 matclayton JoeJulian: Can't see anything relevant there
17:40 semiosis JoeJulian: probably thinking of the old undoc page
17:40 JoeJulian Or... it's now documented...
17:40 JoeJulian at least it is in 3.4...
17:40 asantoni joined #gluster
17:40 semiosis what is?
17:41 JoeJulian cluster.entry-self-heal and/or cluster.data-self-heal
17:42 JoeJulian I don't know what that would do to the self-heal daemon though.
17:42 Theory there's an option in master to only repair data on open, but can't see one to repair it except on open.
17:43 hagarth matclayton: you can use volume set <volname> {data|entry|meta-data}-self-heal off
17:44 hagarth to disable healing from clients
17:44 matclayton on 3.3.1, or is that 3.4.2?
17:44 matclayton hagarth: do I need to set one or all 3 of them?
17:45 hagarth matclayton: same behavior with both 3.3.1 and 3.4.2.  need to set all 3 to disable all 3 forms of self-heal.
17:46 matclayton hagarth: that sounds ideal, will the self heal daemon continue to repair the brick?
17:46 jezier_ joined #gluster
17:46 jiqiren joined #gluster
17:46 hagarth matclayton: in theory, yes it should.
17:46 d-fence joined #gluster
17:47 eryc_ joined #gluster
17:47 matclayton hagarth: lets give this a go then! Thanks I'll let you know if we go off-line again....
17:48 xavih joined #gluster
17:48 rotbeard joined #gluster
17:49 juhaj joined #gluster
17:51 pdrakeweb joined #gluster
17:51 matclayton hagarth: no it still locks up
17:54 asantoni hey guys, I have a bunch of LXC containers that each share a glusterfs mount (specified with lxc.mount). When there's a network interruption, the LXC containers all lose access to the glusterfs mount ("transport endpoint is not connected")
17:55 asantoni Am I doing something wrong strategically here?
17:55 semiosis asantoni: you are either... 1. not using replication, or 2. using replication but clients don't really connect to all replicas
17:55 theron joined #gluster
17:55 semiosis if 2, check name resolution & iptables -- two most common issues
17:56 asantoni I think it might actually be a FUSE issue semiosis though
17:56 asantoni the computer that LXC is running on reconnects fine on its own
17:56 JoeJulian lxc
17:56 semiosis asantoni: well hmm... check the client log file
17:57 asantoni semiosis: ... glusterfs log?
17:57 asantoni k
17:57 semiosis usually /var/log/glusterfs/the-mount-point.lgo
17:57 semiosis log*
18:00 matclayton hagarth: Is it possible to rsync data between replicas, given self heal is taking us offline?
18:00 asantoni semiosis: Should each of the LXC containers be running a glusterfs client directly?
18:01 semiosis asantoni: no idea
18:01 hagarth matclayton: is the data in good bricks being actively modified?
18:01 matclayton not modified, just new files being created
18:02 asantoni ok, thanks for the log file tip semiosis, I'll keep digging
18:02 semiosis asantoni: 'transport endpoint is not connected' is an error that can be diagnosed & corrected... track down the corresponding client log file & get the messages from around that event
18:02 matclayton we're mounted in a worm FS for the most part, and have a daemon deleting some old files right now, but we have good logs of those
18:02 semiosis asantoni: i'd want to rule out the two most likely causes of that error (iptables & name resolution) before looking into more esoteric things like fuse & lxc
18:03 hagarth matclayton: replicate maintains changelogs in the form of extended attributes and is not a good idea to copy over incomplete changelog information from the bricks directly
18:03 asantoni semiosis: well I mean
18:03 asantoni The network went down, came back up, and glusterfs came back fine on the host machine
18:03 matclayton hagarth: those volume settings, does it require a restart of clients or servers to apply?
18:03 asantoni so it won't be a network issue
18:04 * semiosis gbtw
18:04 semiosis too busy to debate.  if you have a log, i'll review it
18:04 hagarth matclayton: no, those settings don't require a restart
18:04 _dist semiosis: I'm back, here is a dpsate of my current test volume ubuntu https://dpaste.de/e7Yz but I assume you want more than just the volume ifo?
18:04 glusterbot Title: dpaste.de: Snippet #256115 (at dpaste.de)
18:04 matclayton hagarth: ok, so they didn't help with locking up, still happened
18:05 matclayton hagarth: we could rsync the xattr's as well.... can you think of any other sensible options...
18:05 glusterbot New news from newglusterbugs: [Bug 1058569] Openstack glance image corruption after remove-brick/rebalance on the Gluster nodes <https://bugzilla.redhat.co​m/show_bug.cgi?id=1058569>
18:07 semiosis _dist: try also performance.flush-behind off
18:07 _dist semiosis: do I need to stop/start the volume each time I make changes like that?
18:07 semiosis _dist: no
18:07 semiosis _dist: check out http://www.gluster.org/community/documentation/in​dex.php/Documenting_the_undocumented#Performance
18:07 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
18:08 semiosis _dist: i suspect one of those performance optimizations is causing your issue, just a hunch
18:08 hagarth matclayton: I need to be afk now, cannot think of anything right away. statedumps might help in identifying deadlocks/frame locks but would have to be exercised carefully.
18:10 _dist semiosis: same issue with flush behind, before I trouble you guys anymore I'm going to try a non-gluster nfsd mount instead to rule out other system issues. Unless you feel certain this is related to gluster
18:17 RedShift joined #gluster
18:18 JoeJulian _dist: I'm fairly sure...
18:19 _dist going through all that nfs4 stuff, might as well be 100%
18:28 Fresleven joined #gluster
18:38 _dist joejulian: same error message using nfsv4, even as a virt-manager storage volume. Now glusterfs is installed but I stopped all the volumes and daemons (had to to get nfs-kernel-server service to own its' port)
18:40 _dist joejulian: it's _possible_ that something glusterfs-server installed is affecting some stage of the migration, but at this point I feel like that'd be a pretty big stretch. However, I have never tried live migration on regular nfs so I might have missed something in the setup. exported as fsid=0 wide open permissions on the img file for both sides (tested that part)
18:44 ells joined #gluster
18:55 _dist joejulian: ok I got my gluster back up and working, and btw you probably saw glusterfs in the netfs setup for virt-manager (that automounts a fuse for you)
18:58 JoeJulian _dist: brb... getting my daughter in a bath...
18:59 _dist no worries, I'm working away at this I'll post anything I discover, I'm going to increase log levels on everything and repeat
19:03 zerick joined #gluster
19:09 theron joined #gluster
19:19 jbrooks joined #gluster
19:32 rcaskey ok, before I plop down the credit card for Azure hosting, any suggestions on VPS providers ;P
19:32 rcaskey (or maybe i should look at leasing a server, granted azure is a 30 day free trial)
19:32 JoeJulian Never met anybody (besides blue networks) that actually uses Azure.
19:33 rcaskey I tested it out a year back and was pretty pleased with it
19:33 rcaskey Was evaluating it for a mysql-ndb cluster
19:34 rcaskey maybe I should just bite the bullet and fedex a box to a data center across the country
19:36 dbruhn what kind of spec's do you need?
19:37 dbruhn I've been using digital ocean here and there for a couple things, they're cheap and fully ssd backed
19:37 samppah rcaskey: https://cloud.digitalocean.com/login we have used this a littble bit
19:37 glusterbot Title: DigitalOcean (at cloud.digitalocean.com)
19:37 samppah dbruhn: nice timing :)
19:38 dbruhn lol
19:38 samppah rcaskey: is azure ready for production use? i heard terrible things about it during beta phase :I
19:39 rcaskey samppah, I just need it for DR
19:40 _zerick_ joined #gluster
19:59 theron joined #gluster
20:05 aixsyd joined #gluster
20:05 aixsyd dbruhn: bro nameth! Any *other* recomendations for decent, used IB switches?
20:06 aixsyd Got that 9024M returned, now looking for an alternative..
20:06 dbruhn did you get to return that one that was wonky?
20:06 aixsyd dbruhn: they gave us some crap about it, but eventually allowed the return
20:06 dbruhn the managed ddr version of that same switch it what I have for my lab
20:07 aixsyd that one was a managed ddr
20:08 dbruhn i thought you had the managed isr?
20:08 dbruhn or sorry the single data rate one
20:08 aixsyd indeed - but it was DDR
20:08 aixsyd nope
20:08 aixsyd definitely DDR.
20:09 dbruhn hmm, well that's the one I have, I bought it because it's what jclift suggested
20:09 dbruhn I use a QDR mellenox in my production stuff though
20:10 aixsyd slightly afraid to get another of the exact same model... but..
20:11 Gluster joined #gluster
20:11 aixsyd http://www.ebay.com/itm/Original-Cisco-​Topspin-120-24-Port-Infiniband-10Gb-Swi​tch-Tops120-/271383788058?pt=US_Network​_Switch_Modules&amp;hash=item3f2fbbfa1a
20:11 glusterbot Title: Original Cisco Topspin 120 24 Port Infiniband 10GB Switch TOPS120 | eBay (at www.ebay.com)
20:16 aixsyd http://www.ebay.com/itm/Flextronics-F-X430046-24​-port-4x-DDR-InfiniBand-Port-Switch-/38083432848​9?pt=US_Network_Switches&amp;hash=item58ab7eafa9
20:16 glusterbot Title: Flextronics F X430046 24 Port 4X DDR Infiniband Port Switch | eBay (at www.ebay.com)
20:31 dbruhn Honestly I have no idea what I would suggest, the biggest thing being there are known card comparabilities in some of the earlier stuff
20:31 asantoni_ joined #gluster
20:32 DV joined #gluster
20:35 aixsyd dbruhn: and I assume I really want a switch with a subnet manager - cause if i use opensm on one node of 5, and that one node goes offline - doesnt the whole fabric go bye-bye?
20:35 Gluster1 joined #gluster
20:36 dbruhn that is the problem indeed
20:36 aixsyd oy. it seems like those 9024's are the only ones in the price range with them
20:37 dbruhn You might be able to run opensm on all your nodes
20:37 dbruhn I am honestly not sure what is involved with running multiple subnet managers on a single network
20:38 Gluster2 joined #gluster
20:39 Gluster joined #gluster
20:40 iksik joined #gluster
20:41 Gluster1 joined #gluster
20:45 Ark_explorys anyone using glusterfs as hadoop namenode metadata NFS?
20:47 Gluster joined #gluster
20:49 badone joined #gluster
20:49 Gluster2 joined #gluster
20:50 andreask joined #gluster
20:51 tru_tru joined #gluster
20:52 Gluster1 joined #gluster
20:53 Gluster3 joined #gluster
21:09 denaitre joined #gluster
21:13 _dist joejulian: so I guess my next step is compile qemu and libvirt tools from scratch. If the issue still exists, I'm worn down enough that I'll just switch distros. If there is any testing you guys want me to do in the meantime I'm up for it
21:15 Gluster joined #gluster
21:19 Gluster1 joined #gluster
21:22 LessSeen joined #gluster
21:25 _dist also speaking of compiling, does anyone have a better solution than equivs to keep my checkinstall stuff from not solving depencies?
21:25 _dist (if you wear a hat instead you'd use a fake rpmbuild) :)
21:25 _dist dependencies*
21:28 semiosis _dist: using ubuntu, right?  what version?
21:29 _dist 13.10
21:29 semiosis cool
21:29 _dist uhh, this is an odd comment I see in the qemu release notes "Migration was broken from QEMU <= 1.5 to 1.6; the new release fixes it. " for 1.7.0
21:30 semiosis are you up for trying to build the qemu/libvirt packages from ubuntu against my gluster packages?
21:30 KyleG joined #gluster
21:30 KyleG joined #gluster
21:30 _dist I've done it, it just requires a source download for the api files
21:31 semiosis i fixed that
21:31 semiosis the latest packages (as of about a week ago) install glfs.h
21:31 _dist by yeah, I'm going to do it today
21:31 ujjain joined #gluster
21:31 semiosis _dist: please let me know how that goes.  i was considering setting up a PPA with such a build, but haven't got around to it yet obviously
21:32 _dist semiosis: did you fix it in you 3.4 ppa?
21:32 semiosis it's not out of the question that ubuntu might include glusterfs in Main so they can build qemu with glusterfs support, but i dont have my hopes up
21:32 semiosis been talking about including glusterfs in Main since precise :)
21:33 semiosis checking
21:33 semiosis oh crap, looks like i only did the debian wheezy packages
21:33 semiosis oops
21:33 semiosis uploading now
21:34 _dist I really like ubuntu honestly, find it easy to work with. I switched from centos back when I first tried out crowbar and it was using it as a default
21:35 _dist but it frustrates me that everything to do with virtualization is a pain :) the main reason I'm running 13.10 is because the kernel is kept low enough to support ZoL
21:37 _dist semiosis: also I've noticed that gluster on 13.10 does not appear to register as a service. glusterd runs at boot time though. There's no /etc/init.d script for it (at least in my installs)
21:38 semiosis hmmm thats odd
21:38 _dist that might be unique to me, but in both my apt-get install glusterfs-server that's the way it ended up
21:38 semiosis it uses upstart, so there should be /etc/init/glusterfs-server.conf (the upstart job) and also a symlink /etc/init.d/glusterfs-server -> /lib/init/upstart-job
21:38 semiosis where are you looking?
21:39 _dist ah, it's just missing the link
21:39 semiosis btw, afaik that symlink only gives you tab completion when you say 'service glusterfs-s[TAB]'
21:39 semiosis if you type out 'service glusterfs-server status' it works
21:39 semiosis hackey stuff
21:39 semiosis upstart </3
21:40 _dist oh that's all it is, I didn't see it in the --status-all and everything I read called it glusterd, glusterfs-server works with the service command fine
21:41 a2 joined #gluster
21:41 gdubreui joined #gluster
21:42 semiosis want to know something else that's funny?  quantal is 6 months older than raring, but quantal is still supported though raring is not
21:42 semiosis support policy change, but still a funny situation
21:43 _dist yeah, it is. I first went to 13.10 instead of 12.04 because I was tired of compiling the new kernel for better pci-stub/vfio support
21:44 semiosis new packages uploaded.  launchpad will build them.  could take minutes or a day
21:45 _dist I have a working win7 gaming guest machine through pci-stub, but I have no idea what hocus pocus made it work (because it's in the same iommu_group as other devices). I tried to do it for a friend with roughly the same hardware but failed.
21:45 _dist semiosis: if it updates soon I'll use it, but probably just download the source :)
21:45 semiosis looks like they should build soon... https://launchpad.net/~semiosis/+arc​hive/ubuntu-glusterfs-3.4/+packages
21:45 glusterbot Title: Packages in “ubuntu-glusterfs-3.4” : ubuntu-glusterfs-3.4 : semiosis (at launchpad.net)
21:45 semiosis https://launchpad.net/~semiosis/+archi​ve/ubuntu-glusterfs-3.4/+build/5553276
21:45 glusterbot Title: amd64 build of glusterfs 3.4.2-ubuntu1~saucy6 : ubuntu-glusterfs-3.4 : semiosis (at launchpad.net)
21:46 semiosis says 7 minutes, which means it should build within the next 60 :)
21:46 _dist I've never had any trouble compiling libvirt 1.2+ or qemu with libgfapi support, the only thing that throws people is the missing glfs.h and the requirement in ubuntu to always use the --prefix in your configure
21:47 semiosis whats the prefix thing?
21:48 _dist well by default it seems most ./configures want to find stuff in /usr/local but ubuntu keeps in them in just /usr
21:49 _dist I've found every time I compile anything before I usually specify the prefix, sysconfdir and localstatedir just incase because different distros have slightly different places?
21:50 semiosis prefix will affect where the software is installed, but you can usually tell it where specific libs are using --with-foo=/path/to/foo.h
21:50 semiosis but ,,(meh)
21:50 glusterbot I'm not happy about it either
21:50 semiosis build packages
21:50 _dist so the qemu configure looks in /usr/local/include/glusterfs instead of /usr/include/glusterfs (could be something I'm doing wrong though). Oh it's the same for where it should go, qemu will put itself in usr/local/bin instead of usr/bin if you don't give it --prefix=/usr
21:51 _dist but if it ends up in usr/local/bin that's not where you want it in ubuntu :)
21:51 semiosis right, there's probably an option you can give like --with-glusterfs=/usr/include/glusterfs so you dont have to set prefix, which means you dont need to let make install write all over /usr
21:51 semiosis well where you want it is in a .deb :)
21:52 _dist you can also change the export, but yeap I just want a deb
21:52 _dist think I should go 1.6.2 or 1.7.0. I'm tempted to do 1.7.0
21:52 semiosis try ./configure --help that should give you a list of all the lib include paths you can set
21:52 semiosis iirc
21:53 _dist yeap, I pretty much always use that, but now things like virt-manager are taking away configure and starting to use python setup.py (it has a --help)
21:55 _dist so about my earlier question, do you know how I can avoid other packages from trying to install qemu-system-x86_64 when I have it compiled myself? I've been using equivs, I keep thinking the checkinstall provides should do the trick but it doesn't
21:55 _dist (equivs is ok, but it's a dirty fix)
21:57 _dist (it probably shows at this point that I've never built a repo, or a ppa) :) I just keep my .debs that I make if they were particularily annoying to make
22:00 _dist semiosis: configure results "GlusterFS support yes" compilation in progress
22:02 semiosis idk jack about checkinstall or equivs
22:03 semiosis only way i would know to do that is to provide the package myself through a ppa
22:03 _dist checkinstall is awesome, it lets you treat compiled packages as through they were installed by a package manager. So you can cleanly remove them etc
22:04 Gluster joined #gluster
22:05 semiosis i guess
22:05 _dist semiosis: well, if you do end up providing a ppa version of qemu & libvirt with libgfapi, that would make plenty of peoeple happy (myself at least anyway)
22:06 _dist ok, compile finished, everything appears to have gone fine
22:06 semiosis my view is that if i have to install software i compile, then i should do that using the distro build process, so i can contribute my work to the distro
22:06 semiosis and only time i've ever really done this (professionally) was/is for gluster
22:06 semiosis most of the time if i can't apt-get install it, i dont use it
22:07 _dist yeah I totally agree, I've just never done it, only compiled locally
22:08 _dist I'll have a .deb in a sec
22:09 _Bryan_ joined #gluster
22:09 pdrakeweb joined #gluster
22:09 _dist ok got it, now I've got to do libvirt 1.2.1
22:12 quique joined #gluster
22:13 quique i have a test pool of three servers and looking at the ports in use in netstat I would expect to see port 49152-49154 in use, but I only see 49152 in use on all nodes.  Why is that?
22:14 quique gluster 3.4.2
22:15 _dist semiosis: this is the output I get from my libvirt configure " glusterfs: yes (CFLAGS='-D_FILE_OFFSET_BITS=64 -D__USE_FILE_OFFSET64 -I/usr/include/glusterfs  ' LIBS='-lgfapi -lglusterfs -lgfrpc -lgfxdr  ')"
22:15 semiosis quique: ,,(ports)
22:15 glusterbot quique: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
22:16 semiosis _dist: idk what you're telling me
22:16 quique semiosis: that doesn't answer my question
22:17 _dist semiosis: sorry just saying my libvirt deb will support glusterfs as a native storage pool through libgfapi
22:17 quique semiosis: gluster doesn't seem to be using an additional port for each brick
22:17 semiosis _dist: oh that's great! \o/
22:17 quique semiosis: i wondering why that is
22:18 semiosis quique: each glusterfs server has a counter, every time you set a new brick on that server the counter goes up (but never down, if you delete a volume or remove a brick)
22:18 semiosis that counter is used to set the port for the brick when it's created
22:18 semiosis at least that's how i understand it
22:18 * semiosis not official
22:18 quique oh the counter is for each brick on each server
22:19 quique i have only one brick on each server
22:19 quique i see
22:19 quique thanks
22:19 semiosis each server has it's own counter, of bricks it has had (even bricks that it no longer has)
22:19 semiosis yw
22:19 semiosis s/it's/its/
22:19 glusterbot What semiosis meant to say was: each server has its own counter, of bricks it has had (even bricks that it no longer has)
22:24 criticalhammer joined #gluster
22:24 chirino joined #gluster
22:25 criticalhammer Hello, does anyone know how much space gluster meta data takes up per file?
22:26 chirino joined #gluster
22:27 chirino joined #gluster
22:28 chirino joined #gluster
22:29 criticalhammer Also, has anyone here know of a good way of separating administrative network traffic onto a separate network?
22:29 chirino joined #gluster
22:30 _dist criticalhammer: do you mean separate gluster commands from gluster storage traffic?
22:30 criticalhammer yes
22:30 chirino joined #gluster
22:30 criticalhammer it would be nice to separate translator traffic from actual data traffic
22:31 chirino joined #gluster
22:31 criticalhammer my googlefu is weak and ive not found any good explanations on this, or if its even possible.
22:31 _dist never thought of it, honestly we keep our entire gluster storage and admin on a separate isolated network. But I can appreciate the point.
22:33 criticalhammer thats what i want to do, but due to ethernet overhead separating command/translator traffic from actual data traffic may have benefits
22:33 kkeithley_ Gluster's metadata is pretty minimal. xfs and ext4, AFAIK, keep xattrs in the inode, and if the available space in the file's inode is filled then it spills over into another inode. This often does happen with Openstack Swift if you don't remember to size your inodes correctly when you create the file system. Your own app can do that too.
22:33 criticalhammer thanks kkeithley
22:34 semiosis criticalhammer: gluster metadata is stored in extended attributes on files.  there's also a .glusterfs folder which has hard links to files.  none of this consumes any of the usable space on the filesystem.  the .glusterfs directory also contains some symlinks (one for each direcotry on the brick) and that will consume a few bytes, equal to the number of characters in all the dir paths concatenated together
22:34 kkeithley_ And if you look on your brick volumes, you'll see the .glusterfs directory, which has a shadow tree full of links to each of the real files. So there's another inode per file and directory
22:34 criticalhammer ok
22:35 criticalhammer does anyone know if its even possible to separate translator/command traffic from data traffic?
22:35 semiosis translator/command traffic?  what does that mean to you?
22:36 criticalhammer clients passing commands to gluster nodes, and gluster nodes performing commands within a gluster cluster
22:36 semiosis have you used glusterfs? :)
22:36 criticalhammer i have
22:36 semiosis clients dont pass commands, they pretty much only do data ops
22:37 dbruhn criticalhammer, you could always run RDMA
22:37 criticalhammer my terminology is off i admit
22:37 semiosis there is some command traffic between servers, for example when you create a volume, but that's miniscule
22:37 semiosis what problem are you trying to solve with this separation?
22:37 criticalhammer what about an ls -l
22:38 semiosis there's no metadata server in glusterfs... the ls -l has to crawl the filesystem on all the bricks just like it would do for a local disk
22:38 criticalhammer not so much a problem, but running different scenarios through my head
22:39 criticalhammer yeah exactly semiosis
22:39 criticalhammer so if i was able to put commands executed by a use on a separate physical network, wouldnt that free up network resources when an end user wants to pull data off a gluster cluster?
22:40 criticalhammer so imagine 2 networks, one that handles commands and administrative traffic
22:40 criticalhammer while another handles purely data transfer between user and cluster
22:40 semiosis cant do it
22:40 criticalhammer ok
22:40 semiosis it's all the same stuff
22:41 chirino joined #gluster
22:41 criticalhammer because ethernet overhead can be really high, i imagine that kind of scenario would increase data transfers
22:41 criticalhammer but yeah its to much seperation
22:41 semiosis that's never come up as a performance bottleneck afaik
22:42 criticalhammer oh ok
22:42 semiosis usually the problem is network latency
22:42 criticalhammer yeah i imagine spinning disks are the bottle neck
22:42 criticalhammer and network latency
22:42 semiosis you can add many spinning disks to a server, and you can add many servers to a switch, but you can't really do much about the latency except switch to infiniband
22:43 criticalhammer yeah
22:43 dbruhn +1 semiosis
22:44 criticalhammer a colleague of mine runs a cluster that uses infiniband, and its not all thats cracked up to be
22:45 dbruhn All of my clusters run infiniband
22:45 criticalhammer not a lot of infiniband tools
22:45 dbruhn if you want to reduce latency it's a beautiful thing
22:45 criticalhammer oh i imagine so
22:45 criticalhammer infiniband is the crem de la crem
22:45 dbruhn A lot of people run IPoverIB
22:46 dbruhn best of both worlds so to speak
22:46 criticalhammer id rather do some type of frame relay, static switching
22:46 criticalhammer in a controlled environment like clusters i dont really see the need for ip addresses
22:46 criticalhammer besides making life easier on the admin
22:47 dbruhn Well that's why we have RDMA
22:47 dbruhn what are you using your cluster for?
22:47 criticalhammer scientific computations
22:48 dbruhn Are your clients all running on the same hardware as your storage?
22:48 criticalhammer the one my colleague runs uses a luster setup for all the scratch storage needs
22:49 criticalhammer the one I run its all nfs to a head node
22:49 criticalhammer err sorry, local scratch and nfs to permanent file storage
22:51 criticalhammer what about you, how do your users use your clusters?
22:52 dbruhn I am running an online backup service, my software run's right on top of my storage nodes.
22:52 dbruhn Gluster helps me reduce my hardware footprint fairly drastically
22:55 criticalhammer what are the specs of your nodes?
22:57 dbruhn Two clusters are Dell r720xd's with 16GB ram per, 2x E5-2609's, with the H710 raid cards, and QDR infiniband. One system is running 12x 600 15k sas, and the other 12x 2tb 7200 rpm sata
22:58 dbruhn the other system I have is some dl180g5's with 410i cards, 24GB of ram per, and 3tb 7200 sata
22:58 sarkis joined #gluster
22:58 dbruhn again qdr infiniband
22:59 dbruhn the dl180's are 12 drives per server too
23:03 _dist semiosis: before I blindly do it, can you explain to me this insecure settings I need to turn on for libgfapi or do you know what they do? I know last time I got the api to work I did need to set them, I think one in the .vol file and one on the volume
23:04 _dist semiosis: right now I get "Transport endpoint is not connected" I remember the solution being those settings
23:04 semiosis _dist: mentioned here: https://github.com/semiosis/lib​gfapi-jni#compiling-and-testing
23:04 glusterbot Title: semiosis/libgfapi-jni · GitHub (at github.com)
23:06 _dist semiosis: got it, I can't beleive I didn't find that page with all my searches :)
23:06 semiosis lol
23:06 semiosis it's virtually unknown
23:06 criticalhammer what does libfapi-jni do?
23:08 semiosis provides jni binding for libgfapi
23:08 _dist semiosis: is service glusterfs-server restart actually enough, or do I need to kill all the jobs and run glusterd? :)
23:08 semiosis _dist: you tell me!
23:09 semiosis afaik that doc i wrote is correct ;)
23:09 _dist semiosis: yeap I will :)
23:20 * semiosis afk
23:21 _dist so libvirt gluster is working, but virt-manager can't do anything with it unfortunately
23:23 khushildep joined #gluster
23:23 _dist which means all xml for drives has to be done by hand, which I guess I have to do to see if it fixes migration :)
23:30 rfortier1 joined #gluster
23:34 daMaestro joined #gluster
23:38 _dist does anyone know the xml for adding a vol into virsh that's stored on a gluster volume attached with libgfapi?
23:40 fidevo joined #gluster
23:43 _dist looks like you can use virsh dump-xml to do it
23:43 _dist what a pain :)
23:49 StarBeast joined #gluster
23:50 _dist I know I did it once, don't remember where I saw the xml example, maybe someone who has one running can paste theirs?
23:51 theron joined #gluster
23:58 andreask _dist: here is an example http://goo.gl/FteUkY
23:58 glusterbot Title: Gluster filesystem users () (at goo.gl)

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary