Camelia, the Perl 6 bug

IRC log for #gluster, 2013-01-23

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 z00dax johnmark: around ?
00:08 jiffe1 joined #gluster
00:08 raven-np joined #gluster
00:58 theron joined #gluster
01:18 sashko_ joined #gluster
01:20 sashko__ joined #gluster
01:31 nik__ joined #gluster
01:35 plarsen joined #gluster
01:54 eightyeight how do i add a glusterfs peer to an existing storage node?
01:54 eightyeight s/node/cluster/
01:54 glusterbot What eightyeight meant to say was: how do i add a glusterfs peer to an existing storage cluster?
01:55 edong23 joined #gluster
02:05 eightyeight got it figured out
02:05 eightyeight that's fairly slick
02:08 eightyeight so, i added to bricks to a replicated distributed volume, and i'm not sure if the replicated data will only be across those two bricks, or across other hosts?
02:08 eightyeight here's a pastebin: http://ae7.st/p/2o8
02:08 glusterbot Title: Pastebin on ae7.st » 2o8 (at ae7.st)
02:10 Ryan_Lane left #gluster
02:11 * eightyeight does a rebalance
02:25 hagarth joined #gluster
02:34 raven-np joined #gluster
02:44 bala1 joined #gluster
02:47 JoeJulian eightyeight: I assume you realized that you have to probe it from the existing trusted pool. A server can't add itself. It's a security thing.
02:48 JoeJulian eightyeight: bricks 7 and 8 (on the same server) will be a replica pair.
02:48 JoeJulian To do what you wanted, you'll need to move a brick to that server (replace-brick) then add-brick, similar to the upgrade process I detailed on my blog.
02:53 Bullardo joined #gluster
02:59 eightyeight JoeJulian: yes. i figured out the adding from a trusted node
02:59 eightyeight JoeJulian: have a link to the post on your blog? i'd prefer that the replication is correct
02:59 eightyeight http://joejulian.name/blog/replacin​g-a-glusterfs-server-best-practice/
02:59 glusterbot <http://goo.gl/pwTHN> (at joejulian.name)
02:59 eightyeight ?
03:01 JoeJulian http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/
03:01 glusterbot <http://goo.gl/BM1qD> (at joejulian.name)
03:01 eightyeight JoeJulian: perfect! thx
03:02 JoeJulian You're welcome.
03:02 eightyeight so, what would you recommend for the order of bricks? eightyeight1,copesetic1,ultraham1,clusterfsck1,​eightyeight2,copesetic2,ultraham2,clusterfsck2 ?
03:06 bharata joined #gluster
03:11 copec Are they going to utilize multicast with ipv6?
03:11 copec that would be cool for replicated writes
03:11 semiosis ,,(brick naming)
03:11 glusterbot http://goo.gl/l3iIj
03:12 copec I guess rpc doesn't use multicast so it would probably get serious work to do that
03:12 semiosis iirc there was some issue with reliability delivery over multicast
03:14 semiosis s/bility/ble/
03:14 glusterbot What semiosis meant to say was: iirc there was some issue with reliable delivery over multicast
03:14 copec I suppose that makes sense
03:15 eightyeight so, if i already added the bricks prematurely, then i'll need to 'remove-brick' them first, then 'replace-brick'
03:16 eightyeight and it appears that i can't remove a brick at a time if replication is 2, which makes sense
03:17 semiosis eightyeight: if you remove a brick, you can't replace it -- it's already gone
03:18 eightyeight semiosis: i get "Brick: clusterfsck:/pool/vmsa/vol already in use" otherwise
03:18 eightyeight # gluster volume replace-brick vms ultraham:/pool/vmsa/vol clusterfsck:/pool/vmsa/vol start
03:18 eightyeight er
03:18 eightyeight typo
03:18 eightyeight heh
03:18 semiosis uh ok i'm confused
03:18 eightyeight nope. i'm good. that's right
03:19 eightyeight semiosis: i'm trying to make sure that clusterfsck, the new peer added, does not replicate data to itself. so, i'm trying to get the brick order correct
03:19 eightyeight currently, the setup is as follows: http://ae7.st/p/2o8
03:19 glusterbot Title: Pastebin on ae7.st » 2o8 (at ae7.st)
03:20 eightyeight following http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/
03:20 glusterbot <http://goo.gl/BM1qD> (at joejulian.name)
03:20 semiosis ok gotcha, seems like you were on the right track then
03:21 eightyeight so, iirc, ultraham:/pool/vmsa/vol should replicate to clusterfsck:/pool/vmsa/vol, and clusterfsck:/pool/vmsb/vol should replicate to eightyeight:/pool/vmsb/vol
03:21 eightyeight but, with the above 'replace-brick' command, i'm getting that error
03:22 hchiramm_ joined #gluster
03:22 semiosis complicated
03:23 eightyeight how to proceed?
03:23 semiosis is starting over an option? :)
03:23 eightyeight heh
03:23 eightyeight it is a sandbox, but a lot of data has been committed.
03:24 eightyeight can i move the data off the bricks before removing?
03:25 eightyeight i guess take the peer out of the cluster
03:26 semiosis eightyeight: tbh, idk... but i would recommend experimenting on a test volume
03:26 eightyeight yeah. that's essentially what this is
03:27 eightyeight no harm if the data is wiped. i don't think anyone has committed anything of importance here
03:28 semiosis fwiw i recommend using the ,,(brick naming) conventions to keep things organized
03:28 glusterbot http://goo.gl/l3iIj
03:28 semiosis ^^
03:29 semiosis also recommend against this replicate-around-a-circle pattern, sure it's possible but hard to maintain with all this careful brick replacement required
03:29 semiosis if you want 2-way replication, add servers in pairs
03:29 semiosis but yeah thats not much help for your current situation
03:30 eightyeight what do you mean 'add servers in pairs'?
03:30 semiosis volume create foo server1:/bricks/foo1 server2:/bricks/foo1
03:30 semiosis peer probe server3
03:30 semiosis peer probe server4
03:31 semiosis volume add-brick foo server3:/bricks/foo2 server4:/bricks/foo2
03:31 eightyeight i see
03:31 semiosis /bricks/foo1 has the same content everywhere it exists
03:31 semiosis same for foo2
03:32 eightyeight this is just for simplicity in administration
03:32 semiosis +1
03:32 eightyeight ok
03:32 semiosis which is very important imho, unless your hardware really costs more than your admins
03:33 semiosis but even then...
03:33 eightyeight definitely
03:33 eightyeight i don't have a hard time understanding the circle design, but there is always a strong argument for simplifying the administration
03:34 eightyeight either way, so long as it's documented, and everyone is on board
03:35 sgowda joined #gluster
03:39 kevein joined #gluster
03:47 raven-np1 joined #gluster
03:50 shylesh joined #gluster
04:03 McLev joined #gluster
04:06 McLev so if I wanted to change the IP address of my gluster volumes...
04:06 McLev in theory, I can just edit the volume files, right?
04:11 semiosis use ,,(hostnames)
04:11 glusterbot Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
04:12 semiosis if you already have bricks it's tricky
04:12 semiosis i suppose you could to a search/replace on vol files but you should stop all clients, stop the volume, and stop the glusterds first
04:19 pai joined #gluster
04:20 Q310 joined #gluster
04:21 Q310 Has anyone here tried using gluster 3.4 for kvm block storage yet?
04:33 McLev yeah, I already setup the bricks. kind of really dumb in hindsight.
04:34 McLev but you guys have never heard of doing something like that and succeeding?
04:36 sripathi joined #gluster
05:03 mohankumar joined #gluster
05:05 hateya joined #gluster
05:24 vpshastry joined #gluster
05:29 sgowda joined #gluster
05:29 melanor9 joined #gluster
05:33 rastar joined #gluster
05:35 raghu joined #gluster
05:44 ramkrsna joined #gluster
05:45 ultrabizweb joined #gluster
05:51 melanor91 joined #gluster
05:51 shireesh joined #gluster
06:05 harshpb joined #gluster
06:07 hagarth joined #gluster
06:15 lala joined #gluster
06:18 sgowda joined #gluster
06:19 sripathi joined #gluster
06:37 ngoswami joined #gluster
06:42 ramkrsna joined #gluster
06:47 RobertLaptop joined #gluster
06:53 JoeJulian McLev: I've heard of people asking the same question. Nobody's reported failure. Many have reported they just recreated the volumes using hostnames instead.
06:56 raven-np joined #gluster
07:00 tru_tru joined #gluster
07:03 raven-np joined #gluster
07:05 Nevan joined #gluster
07:17 sgowda joined #gluster
07:17 jtux joined #gluster
07:18 koodough1 joined #gluster
07:38 sgowda joined #gluster
07:40 sripathi joined #gluster
07:47 jh4cky joined #gluster
07:57 ctria joined #gluster
08:00 andreask joined #gluster
08:04 guigui1 joined #gluster
08:08 jtux joined #gluster
08:09 rgustafs joined #gluster
08:17 Joda joined #gluster
08:22 sahina joined #gluster
08:29 gbrand_ joined #gluster
08:30 gbrand_ joined #gluster
08:33 hagarth joined #gluster
08:33 hagarth @channelstats
08:33 glusterbot hagarth: On #gluster there have been 75594 messages, containing 3372401 characters, 565238 words, 2302 smileys, and 291 frowns; 561 of those messages were ACTIONs. There have been 25847 joins, 931 parts, 24966 quits, 9 kicks, 43 mode changes, and 5 topic changes. There are currently 181 users and the channel has peaked at 203 users.
08:36 pkoro joined #gluster
08:39 dobber joined #gluster
08:44 duerF joined #gluster
08:47 bulde1 joined #gluster
08:47 bulde1 joined #gluster
08:53 DaveS joined #gluster
08:57 glusterbot New news from newglusterbugs: [Bug 903113] memory corruption in rpcsvc <http://goo.gl/QNVvI>
09:02 Nevan1 joined #gluster
09:05 overclk joined #gluster
09:09 bauruine joined #gluster
09:13 smellis joined #gluster
09:18 Norky joined #gluster
09:21 shireesh joined #gluster
09:21 sahina joined #gluster
09:22 pai joined #gluster
09:23 hagarth joined #gluster
09:30 manik joined #gluster
09:37 srhudli joined #gluster
09:41 tryggvil joined #gluster
09:42 bulde joined #gluster
09:48 red_solar joined #gluster
09:55 vpshastry joined #gluster
10:06 Q310 anyone around who is using  gluster 3.3 in prod?
10:10 jmara joined #gluster
10:21 shireesh joined #gluster
10:22 Elendrys joined #gluster
10:24 Elendrys Hi there, Can someone tell me if there is some specificities when upgrading from 3.3.0 to 3.3.1 ? Is it as simple as stop all volumes, stop all process, upgrade packages and restart everything ?
10:24 Shdwdrgn joined #gluster
10:27 sahina joined #gluster
10:31 deepakcs joined #gluster
10:38 Staples84 joined #gluster
10:39 overclk joined #gluster
10:42 duerF joined #gluster
10:44 ndevos Elendrys: yes, it should be that simple, but like any upgrade, you should try it in a test-environment first
10:48 Staples84 joined #gluster
10:55 clag_ joined #gluster
10:55 joeto joined #gluster
10:58 glusterbot New news from newglusterbugs: [Bug 903160] Setattr error logs are seen while self-healing <http://goo.gl/FaO9P>
11:01 rcheleguini joined #gluster
11:10 hagarth joined #gluster
11:30 overclk joined #gluster
11:36 andreask joined #gluster
11:45 spn joined #gluster
11:58 glusterbot New news from newglusterbugs: [Bug 903175] Possible performance issue with md-cache and quick-read <http://goo.gl/dCDwb>
12:05 tjikkun joined #gluster
12:05 tjikkun joined #gluster
12:05 jgillmanjr joined #gluster
12:10 plarsen joined #gluster
12:16 overclk joined #gluster
12:18 raven-np joined #gluster
12:21 RicardoSSP joined #gluster
12:24 hagarth joined #gluster
12:31 nhm joined #gluster
12:31 deepakcs joined #gluster
12:32 hateya joined #gluster
12:32 gbrand__ joined #gluster
12:36 puebele joined #gluster
12:37 dustint joined #gluster
12:37 dustint_ joined #gluster
12:43 jgillmanjr Good day, folks. I seem to have an issue where I'm trying to add nodes/bricks to an existing volume, gluster peer status indicates the new nodes are in the cluster, yet when I try to add them, it will say one of the hosts is not connected.
12:51 aliguori joined #gluster
12:53 rgustafs joined #gluster
12:55 puebele joined #gluster
13:02 Norky do a gluster peer status on all working nodes, and compare the names and UUIDs
13:02 Norky and put the output on a web pastebin service for us
13:03 Norky I am not a glsuter expert, just suggesting basic troubleshooting :)
13:06 balunasj joined #gluster
13:10 duerF joined #gluster
13:13 jgillmanjr Norky: Will do. Give me a moment.
13:21 jgillmanjr Norky: http://pastebin.com/jvcph6if
13:21 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
13:21 jgillmanjr Sometimes it will be gluster05 and sometimes it will be gluster06
13:22 jgillmanjr gluster01 - 04 were the initial bricks upon volume creation
13:23 jgillmanjr I've noticed there are a few different logs in /var/log/glusterfs/ - what would be the best one to check against?
13:25 plarsen joined #gluster
13:29 Norky check in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
13:30 Norky glustershd.log is for the self-healing feature
13:30 edward1 joined #gluster
13:30 Norky i.e. not relevant to this problem
13:31 jgillmanjr k, let me tail -f that while I try adding
13:35 jgillmanjr http://dpaste.org/nqwuC/
13:35 glusterbot Title: dpaste.de: Snippet #217380 (at dpaste.org)
13:36 Norky it might be worth tailf on the 'problem' machines at the same time
13:37 andreask joined #gluster
13:37 jgillmanjr true. I did see "RJT" on lines 25-28 of that previous paste, so I'm curious what might be causing the reject
13:37 jgillmanjr I'll tail gluster05
13:38 Norky the RJTs are all from the new servers
13:39 jgillmanjr correct
13:39 Norky can you try creating a new, small test volume using all 8 machines? It might reveal something more interesting..
13:39 jgillmanjr oh... you know what... I didn't make the /etc/hosts entries..
13:39 jgillmanjr that might be the cause
13:43 abkenney joined #gluster
13:44 jgillmanjr ....aaaandd that fixed it. Much appreciate the help!
13:44 jgillmanjr Norky: My apologies :)
13:45 Norky huh? I woudln't have thought you could even connect the peers without working hostname lookup...
13:45 Norky or do you mean so the new hosts can reverse lookup the IPs of the existing machines?
13:46 Norky I can only say one thing: DNS :)
13:47 jgillmanjr Norky: DNS indeed! This is just test/proof of concept work for now. The issue was that the hosts entries weren't created on the new nodes - which is probably what was causing the rejections
13:47 Norky righto, glad it's sorted
13:50 hateya_ joined #gluster
13:58 lala joined #gluster
14:03 manik joined #gluster
14:10 gbrand_ joined #gluster
14:14 bennyturns joined #gluster
14:19 Azrael808 joined #gluster
14:23 hagarth joined #gluster
14:37 theron joined #gluster
14:41 theron joined #gluster
14:47 stopbit joined #gluster
14:47 rwheeler joined #gluster
14:52 raven-np joined #gluster
15:07 bugs_ joined #gluster
15:10 luis_alen joined #gluster
15:10 lh joined #gluster
15:13 hateya_ joined #gluster
15:16 bennyturns joined #gluster
15:18 luis_alen Hello, guys. Last week I installed gluster and configured a replicated volume on 2 amazon ec2 m1.small instances running with 2.6.32-279.19.1.el6.x86_64. Since this instance only has one core, gluster was not able to spread its cpu load to multiple cores, of course. Well, I then changed the instance type to c1.medium, Now it has two cores but it looks like gluster is only using one, although performance.io-thread-count=64. Should I reinstall the rpm I
15:26 wushudoin joined #gluster
15:33 lhawthor_ joined #gluster
15:35 lh joined #gluster
15:45 Nicolas_Leonidas joined #gluster
15:45 Nicolas_Leonidas hi
15:45 glusterbot Nicolas_Leonidas: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an
15:45 glusterbot answer.
15:46 Nicolas_Leonidas Is there a way for gluster to email admin when something goes wrong?
15:46 jgillmanjr luis_alen: How are you determining how many cores gluster is using?
15:48 andrei_ joined #gluster
15:49 andrei_ hello guys! I was wondering if someone could answer a few questions about the geo-replication?
15:49 andrei_ I am planning to set up a backup server
15:49 andrei_ but the server will not be located in a different physical location
15:50 andrei_ it's going to be in the same rack
15:50 andrei_ should I use the Replicated glusterfs setup or the geo-replicated one?
15:50 andrei_ taking into account that both servers are different in terms of free disk space and performance
15:54 rwheeler joined #gluster
15:58 andreask joined #gluster
15:59 greylurk joined #gluster
16:04 kkeithley Using AFR replication is going to have a performance impact on the clients, and given the size differences of the replica server I'd say you probably shouldn't choose that.
16:05 kkeithley AFR versus geo-rep is really synchronous versus asynchronous replication. (And just because the geo-rep target isn't physically remote doesn't somehow render it unsuitable for geo-rep.)
16:11 Norky Nicolas_Leonidas, rather than being a function of Gluster, I'd suggest that's best done by a monitoring tool, i.e. something that watches log files or polls running processes
16:11 wushudoin left #gluster
16:12 chouchins joined #gluster
16:12 Nicolas_Leonidas Norky: right now I made a script that will inform me if files in the volume are not accessible
16:13 Norky this is Unix/Linux, after all, where the ethos is one tool does one thing, and does it well
16:13 Norky that said, Gluster does have hooks http://www.gluster.org/community/doc​umentation/index.php/Features/Hooks to run arbitrary scripts on certain events, but I think most of those events are 'normal' (i.e. not failure/error) events
16:13 glusterbot <http://goo.gl/1xEGw> (at www.gluster.org)
16:15 Norky that sounds like something you'd want to plug into Nagios
16:17 jgillmanjr Interesting. When I added those new nodes, doesn't look like the storage capacity increased on the client side when I ran df -h (I did rebalance the nodes). So I removed my new nodes (gluster05-08) and readded them using replica 4, same situation. Tried remounting the volume on the client, but no dice.
16:17 luis_alen jgillmanjr: By analyzing top with SMP view and sar. One of the cores stays absolutely idle while two clients write to the gluster volume frenetically.
16:19 Nicolas_Leonidas Norky: thanks
16:19 daMaestro joined #gluster
16:19 andrei_ kkeithley: thanks!
16:19 andrei_ would I be able to set the geo-replicated server as the primary server during system maintenance?
16:20 m0zes joined #gluster
16:20 Norky Nicolas_Leonidas, at the other (crude) end of the spectrum you could run your tool in cron (cron will be default send job output as email)
16:20 Norky jgillmanjr, how many replicas did you have before?
16:21 Nicolas_Leonidas Norky: that's exactly what I'm doing right now
16:22 jgillmanjr Norky: 2. I wasn't sure if the replica count when passed at instantiation of the volume was indicative of the number of nodes in a replicated set, or the number of replicas - thus me trying with the replica 4 when I readded the new nodes
16:22 rwheeler joined #gluster
16:23 jgillmanjr So when I did the initial addition of the new nodes, I saw "Number of Bricks: 4 x 2 = 8" and now with the replica 4 parameter "Number of Bricks: 2 x 4 = 8"
16:23 Norky replica 2 means every object (file) will be stored in 2 places, so you get 50% of the raw space. Doubling the number of bricks but also doubling the replica count means each file will be stored 4 times,
16:24 Norky the first expansion should have worked
16:24 jgillmanjr ok
16:25 jgillmanjr I was thinking it might have been reversed, since when I was removing the nodes from the first attempt, I had to remove gluster05/06 and gluster07/08 in those pairs
16:25 jgillmanjr which is what made me believe that they were in sets of two
16:25 guigui1 left #gluster
16:26 rastar joined #gluster
16:26 Norky download and run https://github.com/fvzwieten/lsgvt for an ascii-art view of brick topology
16:26 glusterbot Title: fvzwieten/lsgvt · GitHub (at github.com)
16:30 jgillmanjr oh, thats quite nifty
16:31 jgillmanjr ok, so actually with the replica 4, this is what i'm seeing: http://dpaste.org/qqScv/
16:31 glusterbot Title: dpaste.de: Snippet #217390 (at dpaste.org)
16:32 Norky yeah, so I would expect you to see only 25% of the raw space
16:33 jgillmanjr hrm, yet the client still shows 300GB when I run df -h
16:34 Norky and each brick is 150GB?
16:34 jgillmanjr That is correct
16:35 luis_alen jgillmanjr: Don't know if you had the chance to read my last message, but I said: "By analyzing top with SMP view and sar. One of the cores stays absolutely idle while two clients write to the gluster volume frenetically."
16:35 jgillmanjr luis_alen: doh, missed that. Let me try something
16:36 luis_alen jgillmanjr: sure
16:38 jgillmanjr luis_alen: I'm going to try and get my issue resolved first, and then I'll resize my instances to see if I get the same result
16:39 luis_alen jgillmanjr: sure. Do you use m1.small as well?
16:41 jgillmanjr luis_alen: I'm actually using Liquid Web's storm product
16:41 Norky jgillmanjr, I think you need to "gluster vol remove-brick test-volume replica 2 gluster0{5..8}:/export05" again, then re-add them
16:42 isomorphic joined #gluster
16:42 Norky n/m, that exact shell glob wont work, because your brick directories have different names (one might ask why? :)
16:42 jgillmanjr Norky: I was starting to ask that question myself lol. OCD, maybe? lol
16:43 Norky jgillmanjr, I think you need to "gluster vol remove-brick test-volume replica 2 gluster05:/export05 gluster06:...." again, then re-add them
16:45 jgillmanjr actually, it will have to be 3,4,7,8 since that's how it looks like the replicated sets were setup as(?)
16:46 Norky ahh, you might not be able to go from "replica 4" straight to "replica 2" - see http://community.gluster.org/q/volu​me-type-changes-supported-in-3-3-0/
16:46 glusterbot <http://goo.gl/7Q5pJ> (at community.gluster.org)
16:46 Norky though I woudl try 4 to 2 first
16:47 Norky I believe gluster will take care of moving the data around appropriately when you shrink, though I suppose (just guessing) it might be faster if you do as you say and save it the rebalancing
16:47 jgillmanjr yeah, it barked at me "Incorrect replica count (2) supplied. Volume already has (4)"
16:47 Norky good test, actually
16:48 jgillmanjr though I didn't do a rebalance
16:50 Norky remove two bricks and specify "replica 3" then remove two more and choose "replica 2"
16:51 Norky do you currently have data on this test volume?
16:51 jgillmanjr a single file - but it is just test data, losing it won't be an issue. However, I suppose this is good experience for working with production environments when changes to the volume need to be made
16:58 zaitcev joined #gluster
16:59 sashko joined #gluster
17:01 Norky it wouldn't hurt to create a few files and see how they are distributed on the bricks
17:03 bitsweat joined #gluster
17:18 lala joined #gluster
17:18 jgillmanjr hrm, running into some other wonkiness now
17:18 jgillmanjr touch: setting times of `file1': Invalid argument
17:18 gbrand_ joined #gluster
17:19 * jgillmanjr ponders starting fresh
17:20 jgillmanjr and actually using /export
17:21 Norky jgillmanjr, you're using distributed-replicated, yes>?#
17:21 rwheeler joined #gluster
17:26 jgillmanjr Norky: That is correct
17:26 Technicool joined #gluster
17:28 Norky is that file1 within a directory, or the root of the volume?
17:30 jgillmanjr That's the thing, it wasn't showing anywhere when I did an ls - on the mount *or* the individual export directories
17:30 jgillmanjr Doesn't matter at this point though (unless I see it come up again) as I'm just starting anew
17:30 Norky by export directories you mean the bricks?
17:31 nik__ joined #gluster
17:31 Norky oo, I have to go home
17:32 Norky have fun with gluster
17:32 jgillmanjr Norky: Correct. Have a great day/evening!
17:33 yesimnathan joined #gluster
17:33 Shdwdrgn joined #gluster
17:37 Mo___ joined #gluster
17:41 rwheeler_ joined #gluster
17:46 yesimnathan Hello!
17:46 glusterbot yesimnathan: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
17:47 yesimnathan I have a replication gluster volume with two bricks.  Only one client is connecting to write/read data.  We've run into the dreaded split brain issue and we're trying to identify why.  Is there a way to mitigate split brain issues when one brick goes off-line and comes back online?
17:50 chirino joined #gluster
18:01 paolo_ joined #gluster
18:03 zaitcev joined #gluster
18:04 gbrand_ joined #gluster
18:14 zaitcev joined #gluster
18:18 rwheeler joined #gluster
18:32 m0zes joined #gluster
18:40 JoeJulian yesimnathan: Depends on what you mean by "off-line". There's several ways to create ,,(split-brain) in the following article though.
18:40 glusterbot yesimnathan: (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://goo.gl/FPFUX .
18:42 JoeJulian yesimnathan: If you're just shutting down one server, doing some stuff, returning that server to service and checking your self-heal info to ensure everything is healed. That shouldn't create any split-brain. If you're shutting down one server, doing stuff, bringing it back and then without waiting shutting down the other server, that could cause it.
18:44 yesimnathan ahh thanks for the info JoeJulian.  We're needing to stop gluster on one of our nodes to trigger snapshots of the volume.  Should stopping glusterd and glusterfsd be sufficiently save assuming the other brick is up the entire time?
18:44 yesimnathan also we're running gluster 3.2.7
18:45 JoeJulian It should, providing you do a ,,(repair) after you return that server to service.
18:45 glusterbot http://goo.gl/uA812
18:45 yesimnathan Excellent.  Thanks so much JoeJulian!
18:45 JoeJulian You're not getting yourself much by stopping the brick, imho, as open files will still be mid-state whether you stop the brick or not.
18:46 yesimnathan ahh.  What would you suggest?  I was originally using remove-brick & add-brick once the snapshot was complete but this caused issues with port assignments for the brick
18:47 JoeJulian There are no really good solutions. I just backup from a client mount, but I have the luxury of being able to schedule quiescent times.
18:48 DrVonNostren is it weird that my gluster volume is showing up as being mounted twice? in both "mount" and df -h to the same location only one entry in fstab....
18:48 yesimnathan I see.  Perhaps we should reinvestigate our backup strategy as we don't have that luxury =P
18:49 JoeJulian DrVonNostren: yep... It's possible you've got it mounted twice. Check /proc/mounts to make sure.
18:49 JoeJulian yesimnathan: The problem is that if a file's open and in mid-write, it's open and in mid-write.
18:50 tryggvil joined #gluster
18:53 DrVonNostren i didnt see anything relelvant in /proc/mounts JoeJulian I unmounted both mounts and am rebooting to see if the double mount happens again
18:53 JoeJulian For my mysql data (which I'm apparently crazy for storing on a clustered filesystem) I do mysqldump(s) pre-backup as a way of checkpointing the state of those dynamic files.
18:53 JoeJulian DrVonNostren: If you don't have double mounts in /proc/mounts - then it sounds like a stale mtab entry.
18:56 Technicool joined #gluster
18:59 glusterbot New news from newglusterbugs: [Bug 903336] Crash observed during io-stats dump using setfattr <http://goo.gl/MMdOR>
19:18 partner would anybody have any idea why the "famous" self-heal trigger doesn't kick in while doing stat <file> does. i can't think any reasons, yet i have the situation at hand.. ?
19:20 partner what i did was (for testing purposes) i went to first node and into brick dir, removed some files (straight from there, not via mount) and wanted to see if it would heal by itself (simulating a situation where somebody would accidentially go and delete something)
19:20 nueces joined #gluster
19:21 partner the volume is simple replica of 2, having 10 GB disks
19:24 partner hmm just noticed this sentence: "In some cases, it may be necessary to read a file's contents for it to be properly replicated"
19:27 portante joined #gluster
19:27 kkeithley partner: what version are you using? On what linux dist?
19:28 partner i just realized i didn't provide that info, sorry
19:28 partner running on debian squeeze with version 3.3.1-1
19:29 yesimnathan JoeJulian: Yeah I think we're just going to have to work on coming up with a different backup strategy unfortunately.  I hadn't considered the mid-write scenario =\
19:31 andrei_ joined #gluster
19:31 luis_alen joined #gluster
19:33 puebele joined #gluster
19:33 Technicool joined #gluster
19:33 DaveS joined #gluster
19:37 gbrand_ joined #gluster
19:38 DaveS joined #gluster
19:40 daMaestro joined #gluster
19:48 partner yeah, can repeat that, i'm just puzzled how "find <gluster-mount> -print0 | xargs --null stat >/dev/null " differs from "stat <gluster-mount>/test.1" - latter fixes while the first doesn't
19:55 chouchin_ joined #gluster
19:57 jbrooks joined #gluster
19:58 ctria joined #gluster
20:04 chouchins joined #gluster
20:04 nik__ anyone know if glusterfs support for freebsd 9.1 is coming any time soon?
20:05 y4m4 joined #gluster
20:08 Technicool joined #gluster
20:11 johnmorr joined #gluster
20:23 bauruine joined #gluster
20:23 m0zes joined #gluster
20:27 tryggvil joined #gluster
20:34 RicardoSSP joined #gluster
20:34 RicardoSSP joined #gluster
20:42 andreask joined #gluster
21:08 rwheeler joined #gluster
21:15 bitsweat left #gluster
21:24 tqrst joined #gluster
21:27 TekniQue joined #gluster
21:29 tqrst any updates regarding the ext4 bug 838784? Our current kernel isn't affected, but we have to update to a version that is most likely affected, and our volume is backed by ext4 bricks.
21:29 glusterbot Bug http://goo.gl/CO1VZ high, high, ---, sgowda, ASSIGNED , DHT: readdirp goes into a infinite loop with ext4
21:57 semiosis migrate your bricks to xfs
21:57 semiosis tqrst:
21:58 semiosis i did that even though i'm not using an affected kernel and things are running a lot smoother
21:58 semiosis i never had any big problems with ext4, but occasionally i would get weird warnings in the log
21:58 semiosis not anymore with xfs though
22:00 glusterbot New news from newglusterbugs: [Bug 903396] Performance improvements related to reduction of system call invocations <http://goo.gl/wiUbE>
22:10 tqrst semiosis: I'd rather not have to migrate 40 bricks
22:11 tqrst the servers' disk slots are full, so I'd have to back everything up over the network * 40
22:38 drockna joined #gluster
22:38 raven-np joined #gluster
22:39 drockna lets say i have 2 gluster volumes. in different data centers. how would i get all the data from one to the other. Then keep then in sync for a short period of time. IE a master/slave situation.
22:39 JoeJulian partner: Yes. That's expected behavior. If you delete something from the brick, the server won't know about it. All the self-heal marking doesn't happen. The only way to fix that is either to lookup() that file, or do a "gluster volume heal $vol full" and wait for it to walk the entire tree.
22:39 al joined #gluster
22:40 JoeJulian partner: Short answer... don't modify your bricks directly. Protect your bricks from unprivileged users gaining access to them.
22:41 hattenator joined #gluster
22:42 JoeJulian drockna: See "geo-replication" in the ,,(rtfm)
22:42 glusterbot drockna: Read the fairly-adequate manual at http://goo.gl/E3Jis
22:43 drockna JoeJulian: Thanks I am looking now.
22:48 manik joined #gluster
22:57 y4m4 joined #gluster
23:03 rwheeler joined #gluster
23:08 manik joined #gluster
23:16 Technicool joined #gluster
23:50 drockna left #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary