Camelia, the Perl 6 bug

IRC log for #gluster, 2013-04-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 phix Are there any issues with running glusterfs to three servers each connected over a VPN?  What happens if the Internet goes down at either location?
00:07 glusterbot New news from newglusterbugs: [Bug 924481] gluster nfs sets unrequested bits in ACCESS reply <http://goo.gl/g6zWA>
00:15 portante joined #gluster
00:22 JoeJulian phix: your latency will be horrid.
00:24 hagarth joined #gluster
00:25 robo joined #gluster
00:25 JoeJulian phix: Also, "what happens" depends on your volume definition. Without replication, the client would see the files on the disconnected servers as missing.
00:34 mtanner joined #gluster
00:41 gluslog_ joined #gluster
00:41 bronaugh joined #gluster
00:44 hagarth joined #gluster
00:56 tjstansell joined #gluster
01:02 chirino joined #gluster
01:04 dbruhn__ joined #gluster
01:08 bala1 joined #gluster
01:11 dbruhn joined #gluster
01:13 d3O joined #gluster
01:15 hagarth joined #gluster
01:17 Shdwdrgn joined #gluster
01:24 d3O joined #gluster
01:37 d3O joined #gluster
01:39 dbruhn joined #gluster
01:41 dbruhn__ joined #gluster
01:48 d3O joined #gluster
01:52 mohankumar joined #gluster
02:07 syoyo_ joined #gluster
02:07 syoyo_ joined #gluster
02:10 jbrooks joined #gluster
02:26 mohankumar__ joined #gluster
02:26 plarsen joined #gluster
02:32 phix JoeJulian: I see, well if I copy a file to server A (via smb/cifs) and the share is pointing to a mounted glusterfs it will live on server A or will it live on any of the servers that the glusterfs volume is associated with?
02:32 phix If I use replication then it will live on all servers but I would have to wait for the file to be copied to all servers before the copy operation unblocks?
02:33 phix if I dont use replication then what? will I still have a latency issue when working with that file on the same site?
02:36 phix If I use replication but one of the servers is down then what?  will the files be synced at a later time? or wont it mount because it is in an inconsistent state?
02:37 phix Please feel free to point me to the appropriate wiki / documentation if this is already covered by them
02:39 phix another question, in replication mode, does glusterfs support a multimaster type replication or does it use a master slave approach?
02:39 phix (I am thinking of using glusterfs for sysvol between three samba 4 servers)
03:40 frakt joined #gluster
03:40 GLHMarmot joined #gluster
03:40 pai joined #gluster
04:02 bharata joined #gluster
04:07 ninkotech joined #gluster
04:07 16WAAQ5DV joined #gluster
04:10 JoeJulian phix: You don't write to bricks, you write to a mounted glusterfs volume. GlusterFS replication is managed by the client (in your case each server would also be a client). Writes to that volume would be done synchronously to all replicas. If those are across a high latency connection, you'll be waiting. As if that wasn't bad enough, before you open a file to read, a self-heal check is done across a replica set. Again, if this is a high latency
04:10 JoeJulian connection, you'll be waiting for that too. High latency peers is just not a good fit for glusterfs.
04:30 pai joined #gluster
04:38 badone joined #gluster
04:59 rastar joined #gluster
04:59 samppah JoeJulian: thanks
05:00 samppah (re: separating the brick)
05:01 kevein joined #gluster
05:02 kshlm|AFK joined #gluster
05:03 kkeithley joined #gluster
05:40 rgustafs joined #gluster
05:41 nueces joined #gluster
05:41 syoyo__ joined #gluster
05:50 samppah JoeJulian: are you still around?
05:53 nitalaut joined #gluster
05:55 nitalaut Hello
05:55 glusterbot nitalaut: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
05:55 nitalaut I'm trying to remove bricks from distributed replicated volume
05:55 nitalaut http://pastebin.com/9GtzBAWV
05:55 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
05:55 nitalaut the error is "Bricks not from same subvol for replica"
05:56 JoeJulian samppah: Yeah, for a few more minutes...
05:56 nitalaut Does anyone know what could be the case ?
05:59 samppah JoeJulian: i'm thinking about reinstall underlying raid.. i guess it's just ok to shutdown glusterfs on that server, reinstall the raid and then let self heal copy the content from another server?
06:00 samppah ie. it's not possible to turn off specific brick?
06:00 JoeJulian That's how I would do it.
06:01 JoeJulian Specific brick: kill the glusterfsd for that brick.
06:02 samppah ok, thanks :)
06:03 JoeJulian nitalaut: For replica 2, each 2 bricks forms a replica subvolume. It's telling you that you're trying to remove two bricks that are in different replica subvolumes.
06:03 bronaugh left #gluster
06:06 Nevan joined #gluster
06:06 nitalaut JoeJulian, I have 2 servers with 2 bricks each. Afaik each server form separate subvolume. I'm trying to remove the second server completely.
06:07 nitalaut What is wrong with this ?
06:07 JoeJulian @brick order
06:07 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
06:08 nitalaut yes
06:08 nitalaut I understand this
06:08 JoeJulian You can do what you're trying to do by adding "replica 1" to your remove brick command. You will no longer have a replicated volume at that point.
06:08 nitalaut Brick1: gw01:/raid, Brick1: gw01:/raid, Brick3: gw01:/mail, Brick4: gw02:/mail
06:08 nitalaut here's what I have
06:08 nitalaut aha
06:08 nitalaut thanks )
06:09 nitalaut that's what I wanted to do
06:09 JoeJulian Good thing I can read minds. ;)
06:09 nitalaut )))
06:11 nitalaut hmm, I don't see such option in help
06:11 JoeJulian What version are you running?
06:11 nitalaut should it be something like 'gluster volume remove-brick mail gw02:/mail gw02:/raid replica 1'
06:11 nitalaut 3.2
06:11 JoeJulian That's why
06:11 nitalaut )))
06:11 nitalaut so sad
06:12 JoeJulian You have two options. Upgrade or delete the volume and recreate it.
06:12 nitalaut ))
06:14 nitalaut JoeJulian, thank you for you help
06:14 JoeJulian You're welcome.
06:19 phix hai
06:19 phix are we all awake now?
06:21 phix JoeJulian: hmmm I see, so even if I use replicas or just use distribute I will be slowed down by network latency?
06:21 JoeJulian Distribute only will be less effected, but you'll be pulling from remote servers 1/3 of the time.
06:22 phix JoeJulian: My connection is ADSL which has a max upload of 1Mbit and max download of about 10Mbit (that is the average that all sites sync at), pinging each site accross the VPN yields, on average, 26ms
06:22 phix is this going to cause a huge issue or not?
06:22 JoeJulian Only 1 way to find out.
06:23 phix How does distribute work?  when you save something to the mounted glusterfs does it do some type of round robin to determine which server to save the file on?  or does it save the file on the server you are copying the file to?
06:24 JoeJulian http://joejulian.name/blog​/dht-misses-are-expensive/
06:24 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
06:25 phix heh, a self proclaimed all round cool guy :P
06:25 JoeJulian :D
06:26 ninkotech_ joined #gluster
06:26 phix heh nah I reckon you are quite cool too :P
06:26 JoeJulian I try
06:26 phix have you worked with samba4 at all?
06:26 JoeJulian No
06:27 JoeJulian I probably will, but I'll hate myself for doing it.
06:27 phix heh, there is nothing wrong with samba4, it aims at replacing windows servers with linux / unix boxes, I am all for that
06:27 JoeJulian But it means you're still supporting windows boxes.
06:28 phix yes however convincing people to use something else is sometimes impossible, at least samba4 is a nice stop gap messure :)
06:28 JoeJulian I know.. I keep whittling away at them.
06:28 phix especially when you have to deal with people that want to use windows in a networked environment that you manage
06:29 JoeJulian It's easiest when someone moves on from the company. They leave, their machine gets formatted with linux.
06:29 JoeJulian New employee's told that's what they're using.
06:29 phix any way, I want to replicate the sysvol (group policy) share between multiple servers, currently I am just rsyncing and enforcing that all group policy changes happen on a "master" server, but would be nice to use glusterfs or the like to distribute it
06:30 phix how does glusterfs go with locking?
06:30 JoeJulian Unfortunately, I can't convince enough people that they should move on... ;)
06:30 phix Are you a Ubuntu / mintOS fan or a CentOS fan?
06:30 JoeJulian CentOS.
06:30 phix (for desktop)
06:30 hagarth joined #gluster
06:31 JoeJulian Fedora for myself, but CentOS for my users.
06:31 phix fair enough
06:32 phix I have had to use CentOS servers and they were a pain, probably because they were using an old version of it (5.4 or something)
06:32 JoeJulian Ok, so... how would I do it... Hmmm
06:32 phix yes, hmmm
06:33 JoeJulian There's really not that much in sysvol... you don't need roaming profiles?
06:34 ollivera joined #gluster
06:41 phix I use roaming profiles and I map appdata to the users homedir on the server
06:41 phix I don't really want to replicate /home or /srv/samba/profiles, that needs to be latency free
06:41 JoeJulian Whew...
06:41 JoeJulian Then sure, I'd try it.
06:42 phix atm I just want to try sysvol, however speaking to one of the samba devs there may be an issue with locking
06:42 JoeJulian Perhaps. I know I disabled all oplocks.
06:43 phix yeah, I should probably do that in sysvol as a starter
06:43 JoeJulian But I'm also running an old version.
06:43 phix samba 4 is stable now, it is awesome
06:44 phix I have been using it for nearly a year now, even in its beta form it was usable, I set it up for a client that have 3 sites / offices in different geo locations
06:45 phix there were some teething problems with saving roaming profiles back to the server and replication but updating to last octobers beta fixed that up
06:49 d3O joined #gluster
06:49 guigui3 joined #gluster
06:50 hagarth joined #gluster
06:50 vimal joined #gluster
07:05 ctria joined #gluster
07:05 tjikkun_work joined #gluster
07:18 ricky-ticky joined #gluster
07:25 spider_fingers joined #gluster
07:26 tjstansell left #gluster
07:28 acalvo When uploading large files I'm getting a ChunkWriteTimeout (10s), however it does work with smaller files. I've deployed just 1 server with both the gluster and the swift.
07:31 ProT-0-TypE joined #gluster
07:36 andreask joined #gluster
07:40 hybrid512 joined #gluster
08:16 gbrand_ joined #gluster
08:16 ujjain joined #gluster
08:20 gbrand___ joined #gluster
08:38 joehoyle joined #gluster
08:53 Chiku|dc joined #gluster
09:00 Chiku|dc what this features.lock-heal really does ?
09:00 Chiku|dc with volume set help, I see it as off
09:00 Chiku|dc but on the 3.3.0 guide is on by default
09:08 rastar joined #gluster
09:09 Chiku|dc JoeJulian, when there is no entrie with gluster volume heal $VOLUME info split-brain, doesn't mean there is no split-brain ?
09:15 clag_ joined #gluster
09:16 hybrid512 joined #gluster
09:18 lalatenduM joined #gluster
09:19 mjrosenb [2013-04-11 05:18:10.258407] I [client-handshake.c:1445:client_setvolume_cbk] 0-magluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
09:19 glusterbot mjrosenb: This is normal behavior and can safely be ignored.
09:19 mjrosenb glusterbot: thanks!
09:19 glusterbot mjrosenb: I do not know about 'thanks!', but I do know about these similar topics: 'thanks'
09:19 mjrosenb glusterbot: thanks
09:19 glusterbot mjrosenb: you're welcome
09:21 mjrosenb [2013-04-11 05:18:10.318834] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
09:21 mjrosenb is that also normal (i'm not used to seeing two versions on one line like that)
09:21 hagarth joined #gluster
09:30 hybrid512 joined #gluster
09:32 spider_fingers glusterbot: pour me some tea
09:33 dobber_ joined #gluster
09:36 Chiku|dc I try to clear-locks but I don't undestand the range
09:36 Chiku|dc for inode
09:45 manik joined #gluster
09:50 Chiku|dc what is the meaning of 0,0-0
09:51 spider_fingers glusterbot: hello
09:55 Chiku|dc someone already clear-locks ?
10:00 manik joined #gluster
10:02 Chiku|dc ok I juste ignore the range, now my file is heading
10:02 Chiku|dc but I still don't know how to use the range
10:10 hybrid512 joined #gluster
10:23 rotbeard joined #gluster
10:32 H__ upgraded another 3.2.5 to 3.3.1 but none of the brick glusterfs start.  I see "E [glusterd-store.c:2080:glus​terd_store_retrieve_volume] 0-: Unknown key: brick-0" but volume info shows the bricks just fine. Any ideas ?
10:42 ollivera joined #gluster
10:45 H__ found another /var/lib/glusterd/ with different UUIDs. That one works.
10:57 joehoyle joined #gluster
11:09 edward1 joined #gluster
11:33 zykure|uni hey, any idea why my distributed setup (2 hosts) is only half as fast as the one-brick setup?
11:39 Chiku|dc After I clear-locks, one of my replicated server is not as befoer, it just wait for self heal only
11:40 mynameisbruce joined #gluster
11:40 mynameisbruce_ joined #gluster
11:40 Chiku|dc how to fix this server ?
11:40 mynameisbruce left #gluster
11:41 piotrektt_ joined #gluster
11:42 Chiku|dc and there is a folder .landfill
11:44 juhaj I'm curious, is there an android or meego client for glusterfs?
11:56 andreask joined #gluster
12:01 bennyturns joined #gluster
12:07 kkeithley1 joined #gluster
12:07 lh joined #gluster
12:28 H__ juhaj: not that i know of
12:42 johnmark juhaj: there's an html 5 app for UFO - https://github.com/joejulian/ufopilot
12:42 glusterbot Title: joejulian/ufopilot · GitHub (at github.com)
12:42 johnmark but other than that, no
12:42 * johnmark would LOVE to have one
12:42 andrewjs1edge joined #gluster
12:45 jskinner joined #gluster
12:51 NeatBasis_ joined #gluster
12:59 acalvo joined #gluster
12:59 acalvo Is it necessary to tweak some swift configuration in order to allow uploading large files?
13:00 JoeJulian mjrosenb: Yep, that's normal.
13:00 robos joined #gluster
13:00 RobertLaptop joined #gluster
13:01 JoeJulian ~pasteinfo | zykure|uni
13:01 glusterbot zykure|uni: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
13:02 chirino joined #gluster
13:02 JoeJulian Chiku|dc: What is the problem you're trying to solve?
13:09 zykure|uni JoeJulian: http://fpaste.org/U4CI/
13:09 glusterbot Title: Viewing Paste #291191 (at fpaste.org)
13:10 zykure|uni oh i put a mistake in there, use this one: http://fpaste.org/UNoK/
13:11 JoeJulian That second one gives me a 500 error.
13:11 zykure|uni hm, seems like fpaste doesn't like the second one
13:11 dustint joined #gluster
13:11 zykure|uni the use the first please
13:11 zykure|uni i just have a typo in the hostname in "volume status" (ibhost1->ibhost2)
13:13 JoeJulian ok...
13:13 zykure|uni i'm pretty sure the data is transmitted over the correct interface (inifniband), checked with iptraf and ifconfig
13:14 JoeJulian Is your client one of the servers, or a third machine?
13:14 zykure|uni a third machine
13:14 zykure|uni i also checked on both the servers though
13:14 zykure|uni even on the servers it's less than what the one-brick volume achieves
13:15 zykure|uni so i noticed the file is written to the second host (i was just using dd with 512M testfile)
13:15 JoeJulian Do you get the same result regardless of which server is the single?
13:15 zykure|uni hm i didn't check that
13:16 zykure|uni only created the single on one server
13:16 \_pol joined #gluster
13:16 JoeJulian I'm just trying to isolate the problem. It shouldn't be any slower. Should be the same speed either way, I think.
13:16 zykure|uni interestingly, if i access the volume on each of the servers i get different results
13:17 zykure|uni like, on ibhost2 it#
13:17 zykure|uni .. it's 500 MB/s
13:17 zykure|uni on ibhost1 only 250 MB/s
13:18 zykure|uni but that could be an effect of where the actual data is stored (i.e. in which of the two bricks)
13:19 JoeJulian right
13:19 a2 joined #gluster
13:22 H__ known causes for this ? -> gluster volume status operation failed Failed to get names of volumes
13:22 hagarth joined #gluster
13:25 zykure|uni JoeJulian: i get about the same result if the sinlge brick is hosted on the other server
13:25 H__ restarting all glusterd made 'gluster volume status' work again
13:26 zykure|uni uh, "same result" is not intuitive
13:26 zykure|uni it's the same speed as the two-brick volume
13:26 zykure|uni and the single brick on the first host is twice as fast
13:27 alex88 joined #gluster
13:31 zykure|uni the local volumes achieve rates of 2 GB/s on the first, and 1 GB/s on the second host
13:32 zykure|uni could that be related? but it's still much more than one you get over the exported glusterfs volume on the clients
13:32 zykure|uni s/than one/than what/g
13:32 acalvo join #swift
13:32 glusterbot zykure|uni: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
13:33 H__ Question belongs here : Can one lower the source brick IOPS load that follows from a replace-brick by rsyncing the source brick directly to the destination brick before starting replace-brick ?
13:33 JoeJulian zykure|uni: So from your 3rd client, you get speeds 4x as fast using a single brick volume from either server.
13:34 JoeJulian H__: Wouldn't rsync produce the same iops?
13:37 zykure|uni JoeJulian: uhm no, on the client it's either the same or half the speed as on the server
13:37 zykure|uni damn i should write that down, it's becoming confusing :)
13:38 H__ it's much lower, and does not hang glusterfs for me while replace-brick does. See bug 950024
13:38 glusterbot Bug http://goo.gl/RBGOS urgent, unspecified, ---, kparthas, NEW , replace-brick immediately saturates IO on source brick causing the entire volume to be unavailable, then dies
13:38 JoeJulian A client mount on the server in a distributed volume is only going over the network for half the files. That result would be irrelevant.
13:38 JoeJulian H__: that's nasty.
13:39 JoeJulian H__: It's a replicated volume, right?
13:39 H__ tell me about it :-( 20 minutes production outage, while i tested this thing on several test setups without issues
13:39 H__ yes, Distributed-Replicate
13:40 JoeJulian What about doing a "commit force" and letting self-heal handle it?
13:40 H__ I don't dare that
13:41 H__ because it is no longer HA during its most critical operations then
13:41 nueces joined #gluster
13:41 bennyturns joined #gluster
13:41 H__ in a test setup : fine, in production : hell no ;-)
13:42 H__ I need to slow down replace-brick so that the volume stays responsive.
13:42 JoeJulian I told pranith the same thing. :D
13:43 zykure|uni JoeJulian: i wrote my results down: http://fpaste.org/78iK/
13:43 glusterbot Title: Viewing Paste #291196 (at fpaste.org)
13:43 zykure|uni the first column is where the tests were performed
13:45 JoeJulian Seems like a pretty clear picture that host1 is slower. I'd start isolating hardware and fix it. :D
13:45 chirino joined #gluster
13:46 hagarth joined #gluster
13:47 Nagilum_ joined #gluster
13:48 Chiku|dc JoeJulian, hi
13:48 Chiku|dc I fixed my file which was locked
13:49 JoeJulian Oh, good.
13:49 Chiku|dc with clear-blocks
13:49 Chiku|dc and the file healed good
13:50 Chiku|dc but the gluster server which this file locked not just not like better
13:50 Chiku|dc it seems not sync replicated, but every 10minute just self heal from the 2nd server
13:51 Nagilum_ I'm having trouble starting glusterd on a node that part of a gluster, it says it starts up ok but the exits, the etc-glusterfs-glusterd.vol.log says: http://sebsauvage.net/paste/?a11192fbb73287c2​#DkaWSUVF3O6Shdl1GpfF3BYfT1IlcAsw2VBjEckh1B8=
13:51 glusterbot <http://goo.gl/Dj94l> (at sebsauvage.net)
13:52 * JoeJulian grumbles about javascript being required to view that.
13:53 lh joined #gluster
13:53 lh joined #gluster
13:54 Nagilum_ JoeJulian: it was too big for fpaste and pastebin is shunned upon here..
13:54 JoeJulian Chiku|dc: Check the glustershd logs and the ,,(extended attributes) on that file on the bricks.
13:54 glusterbot Chiku|dc: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
13:54 H__ JoeJulian: do you perhaps know if seeding a replacement brick would work at all ?
13:55 zykure|uni JoeJulian: but why is the single brick faster even when the data is distributed to the same host in the two-brick volume? :)
13:55 JoeJulian That was too big for fpaste??? It's tiny. :P
13:55 JoeJulian H__: Sure. It'll work.
13:58 Nagilum_ JoeJulian: thats what it told me..now it works, who knows
13:58 H__ would it also lower replace-brick's workload ? or does that involve reading the entire source brick anyway ?
13:58 JoeJulian zykure|uni: Dunno... Doesn't seem like there should be any computational overhead.
13:59 JoeJulian H__: I've never tried that so I couldn't say.
13:59 H__ ok, I'll test it then.
13:59 JoeJulian H__: I'd consider doing it that way and still doing the "commit force".
14:00 H__ reason i'm asking first is that testing with terabytes takes days ;-)
14:00 JoeJulian I hear ya.
14:00 H__ hmm, that' s interesting.
14:00 JoeJulian Can you set up a test environment to make sure that idea's even valid?
14:00 H__ yes, i have 3 test setups
14:00 Nagilum_ JoeJulian: is anything I can do to bring that node back into the gluster?
14:01 JoeJulian H__: Oh, good.
14:01 Nagilum_ JoeJulian: as far I can see the files in /var/lib/glusterd look ok
14:01 H__ well, 2 actually. the 3th is production hot-spare ;-)  I use it as acceptance testbed :-D
14:01 JoeJulian Nagilum_: Were we talking about this a couple days ago?
14:02 Nagilum_ JoeJulian: no
14:03 JoeJulian Ah, right, that was dbruhn...
14:04 Chiku|dc what about the folder .landfill ?
14:04 hagarth joined #gluster
14:04 Chiku|dc there is this folder in this server
14:05 Chiku|dc reading about xattrib
14:05 JoeJulian I know that .landfill exists, but I'm not really sure what it's used for.
14:06 JoeJulian Nagilum_: Try starting with glusterd --debug and paste that output.
14:06 Nagilum_ k
14:09 zykure|uni JoeJulian: okay thanks for your help again, i'll see if i can find out more about this
14:10 iatecake joined #gluster
14:10 Nagilum_ http://sebsauvage.net/paste/?0bb06857e8a33881​#GmhEEaqvEUAX8k6HF5g0elu0MVLt9sbDkaoVE0mOHEI=
14:10 glusterbot <http://goo.gl/cIQvo> (at sebsauvage.net)
14:12 JoeJulian Nagilum_: Check the glusterd log on g9t3041 and see if there's some clue.
14:15 Nagilum_ hmm: http://sebsauvage.net/paste/?3a5a26cc9a75c584​#D2WYq/ipHJRr3Xio2WsdLa5w1MXJqFIqLtvbkc/2w8c=
14:15 glusterbot <http://goo.gl/nkWDI> (at sebsauvage.net)
14:16 Nagilum_ will restart glusterd on 41
14:16 spider_fingers left #gluster
14:16 JoeJulian Nothing right before that though...
14:18 Nagilum_ [2013-04-11 14:07:15.847689] W [socket.c:195:__socket_rwv] 0-management: readv failed (Connection reset by peer)
14:18 JoeJulian I was hoping something led up to the disconnect.
14:19 iatecake joined #gluster
14:19 gbrand_ joined #gluster
14:19 brunoleon_ joined #gluster
14:19 sefz joined #gluster
14:19 stoile joined #gluster
14:19 msvbhat_ joined #gluster
14:19 mriv joined #gluster
14:19 helloadam joined #gluster
14:19 shawns|work joined #gluster
14:20 Nagilum_ JoeJulian: there just tons of "I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now"
14:20 JoeJulian weird.
14:21 manik joined #gluster
14:21 JoeJulian Can you share a tcpdump from the server that's not starting?
14:21 JoeJulian Just port 24007 should be sufficient.
14:22 Nagilum_ k, give me two mins
14:22 hagarth joined #gluster
14:29 tjikkun_work i get a backtrace in the logs: http://fpaste.org/0t3G/
14:29 glusterbot Title: Viewing Paste #291222 (at fpaste.org)
14:30 rastar joined #gluster
14:30 tjikkun_work where should I submit this?
14:30 JoeJulian Glusterbot will tell you where to file a bug
14:30 tjikkun_work my mountpoint at that point says Transport endpoint is not connected
14:30 JoeJulian file a bug
14:30 glusterbot http://goo.gl/UUuCq
14:31 tjikkun_work JoeJulian, thanks
14:34 JoeJulian Nagilum_: Ah, so the RST is not the cause. :(
14:35 bugs_ joined #gluster
14:36 joeto joined #gluster
14:36 Nagilum_ is there no easy way to bring the node back into the gluster? deleting some and then doing "volume sync g9t3038" or something?
14:36 robos joined #gluster
14:36 Nagilum_ s/some/some file/
14:36 glusterbot What Nagilum_ meant to say was: is there no easy way to bring the node back into the gluster? deleting some file and then doing "volume sync g9t3038" or something?
14:37 JoeJulian We could try deleting /var/lib/glusterd/vols and see if that works.
14:37 JoeJulian It won't hurt anything
14:37 Nagilum_ let me move it away..
14:38 Nagilum_ no change
14:41 Nagilum_ hmm
14:41 Nagilum_ two files in the peers directory are empty
14:42 Nagilum_ let copy them from another node
14:42 Nagilum_ +me
14:42 JoeJulian I was just about to go there...
14:43 Nagilum_ ha!
14:43 Nagilum_ its running again
14:44 JoeJulian Lame... Someone should file a bug report on that.
14:44 glusterbot http://goo.gl/UUuCq
14:46 JoeJulian Yep, that's easy to repro. Not sure how the peer file can be empty, but that's a good bug.
14:52 sefz joined #gluster
14:58 rwheeler joined #gluster
15:02 jdarcy joined #gluster
15:10 zykure joined #gluster
15:10 glusterbot New news from newglusterbugs: [Bug 951177] glusterd silently fails if a peer file is empty <http://goo.gl/DLrVR>
15:10 Nagilum_ :)
15:12 sefz Hi, i'm doing some test, i'm experiencing very high speed reads (700MB/s) but very low disk writes (around 10MB/s), and I wanted to do more tests... I tried to create a  big file using dd on the gluster mountpoint, but I reiceive this: dd: opening `/mnt/gluster/dd_test': Invalid argument
15:12 rotbeard joined #gluster
15:13 dxd828 what is the difference between glusterd and glusterfsd? I have killed off glusterd and my clients can still write data to the bricks
15:13 semiosis dxd828: ,,(processes)
15:13 glusterbot information.
15:13 glusterbot dxd828: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more
15:13 hagarth joined #gluster
15:14 dxd828 semiosis, thanks!
15:14 semiosis sefz: pastie the client log from when you got the invalid argument error.  also, dd is a horrible benchmarking tool.  if you insist on using it, then you should use bs=1M or larger when writing into glusterfs, but it's still not good.
15:15 semiosis sefz: dd could be made more useful as a benchmarking tool for glusterfs if you were to run lots of dd in parallel
15:15 semiosis s/lots/several/
15:15 semiosis depending
15:15 glusterbot semiosis: Error: I couldn't find a message matching that criteria in my history of 1000 messages.
15:16 semiosis glusterbot: why do you hate me?
15:16 michael_1 joined #gluster
15:16 Supermathie ndevos: Oohhhhh, nice catch on the FSINFO RPC.
15:16 Supermathie Trying it out now.
15:17 ndevos Supermathie: well, I did not spend too much time on it, so dont expect too much from it
15:18 * ndevos does not like gfid mismatches btw
15:20 zykure joined #gluster
15:20 Supermathie Some warnings: http://pastie.org/7454498
15:20 glusterbot Title: #7454498 - Pastie (at pastie.org)
15:20 semiosis ,,(gfid mismatch) is back?
15:20 glusterbot http://goo.gl/uoyTN
15:20 Supermathie ndevos: Finding the source of the problem is key :)
15:21 semiosis JoeJulian: is glusterbot mining bitcoins in the background or is it just me?  seems like the bot's been lagging this week
15:22 ndevos semiosis: well, we've found a way to trigger a gfid mismatch... not a very common situation, but its possible and reproducible :-/
15:22 ndevos if I only understood more details on whats being logged
15:23 Nagilum_ semiosis: bc mining is done via the glusterfsd's for scalability reasons :>
15:24 semiosis ndevos: bug id?  also i thought there was automatic gfid mismatch repair code in glusterfs since like 3.2.6 or something
15:24 ndevos semiosis: filing one now
15:24 semiosis ok i'll check it out when you're done thx
15:28 dbruhn joined #gluster
15:28 glennpratt joined #gluster
15:33 hagarth joined #gluster
15:38 Supermathie ndevos: that broke Linux's NFS :P
15:38 xiu joined #gluster
15:38 ndevos Supermathie: hehe
15:39 Supermathie [2013-04-11 11:38:11.283020] E [nfs3.c:4741:nfs3svc_fsinfo] 0-nfs-nfsv3: Error decoding arguments
15:39 Supermathie might do: if (original fails) { (new code) } ...
15:39 ndevos thats ugly!
15:39 xiu hi, in io profiling, what does the duration means ?
15:40 ndevos Supermathie: I'm not sure what NFS standard says about an opaque value, maybe it should be xdr_bytes() and your Oracle NFS client is misbehaving?
15:41 Supermathie ndevos: As far as I'm concerned, if Oracle is breaking the XDR spec that's a bug on their end. However, I think gluster should handle it better due to the 'be liberal in what you accept' mantra
15:44 Supermathie if (&objp->data.data_len % 4) { pad_it_with_zeroes; } // :)
15:45 semiosis postel's law
15:45 H__ I see about a factor-10 slowdown for rsync crawling a gluster tree after the upgrade from 3.2.5->3.3.1
15:45 ndevos Supermathie: well, thats the xdr specification way, oracle needs the else-clause
15:46 ndevos Supermathie: if your oracle client works now, it may be worth to check how they xdr-encode it, and compare that to the linux nfs-client, I do not know which is really correct
15:50 Supermathie ndevos: I'll just point this out: Oracle DNFS works Just Fine against NetApp, Solaris, Linux, etc... (doesn't mean oracle is right, just means those others are handling this case nicer)
15:50 Supermathie ndevos: Well, Linux is padding it, what you point out all makes sense... it sounds right.
15:51 robo joined #gluster
15:52 ndevos Supermathie: actually, I'm surprised the xdr_opaque() failed, but maybe the function that decodes the nfs_fh3 could indeed append/discard those extra fill-bytes
15:53 ndevos Supermathie: I think such padding is done elsewhere in the xdr conversion code, maybe you find some examples for it
15:57 saurabh joined #gluster
15:59 Supermathie ... probably should have stopped the Oracle DBs before I stopped NFS ...
16:01 jclift_ joined #gluster
16:02 * sefz idle for 20 minutes: starting autoaway 14(15m1rk 15Xp)
16:04 ndevos semiosis: bug 951195 is the one I just filed
16:05 glusterbot Bug http://goo.gl/7QR9p high, high, ---, sgowda, ASSIGNED , mkdir/rmdir loop causes gfid-mismatch on a 6 brick distribute volume
16:07 zonkrat joined #gluster
16:11 zonkrat Hello all.  We're trying to determine the best approach to fixing this bug ((https://bugs.launchpad.net/swift/+bug/1065869).   When our application is processing thousands of http GETs that include byte ranges we're seeing major breakage in the swift object server error log.    We're using the Redhat/EPEL gluster 3.3 rpms on CentOS 6.4.   Given the code level we're at, might there be any suggestions on wether to patch this code (assuming patches exist)
16:11 glusterbot Title: Bug #1065869 “ranged GETs always perform two backend GETs” : Bugs : OpenStack Object Storage (swift) (at bugs.launchpad.net)
16:20 Supermathie ndevos: Nope, doesn't change anything
16:20 Supermathie ndevos: We can step through it in gdb
16:21 ndevos Supermathie: hmm, I'm finished for the day and will be gone soon, can you post a new tcpdump? maybe the MNT Reply is broken now?
16:22 Supermathie ndevos: tcpdump looks exactly the same. I modified your patch to be:
16:22 Supermathie I suppose I could open a formal RHEL ticket for this... ugh red tape
16:23 Supermathie (actually, will email it)
16:23 hybrid5121 joined #gluster
16:27 ndevos Supermathie: if you open a RHEL (at Red Hat) ticket for it, let me know the number and I'll check whats happening from that side ;)
16:27 * ndevos <- gone for the day
16:30 dxd828 Can you have multiple vol's per bricks? Or do I need to create a sub directory for each one, or is that bad?
16:35 semiosis dxd828: brick is a directory on a server
16:35 semiosis dxd828: you can have for example /mnt/disk a mounted block device, then /mnt/disk/vol1 and /mnt/disk/vol2 as two bricks (each in a different volume) on that mounted disk
16:35 semiosis if you want
16:36 semiosis although you'll have strange disk capacity reports... for example in df the available/used/total may not add up -- i'm not sure about that but it seems possible
16:38 Supermathie I could see the case where you might want to have >1 brick on a single FS... one for a distributed-replicated volume, one for a striped-replicated volume.
16:38 Supermathie Buuuuuuut that's kind of odd.
16:38 Mo___ joined #gluster
16:38 dxd828 semiosis, ok it is just atm I have my brick is just a mounted drive, and I want to have multiple vol's on that disk
16:41 dxd828 semiosis, just thinking about best practice should I keep a single brick for my drive then just use sub directorys for different things on my clients?
16:57 bala joined #gluster
17:08 tryggvil joined #gluster
17:10 6JTAAD80S joined #gluster
17:21 _pol joined #gluster
17:21 _pol joined #gluster
17:30 robos joined #gluster
17:38 nueces joined #gluster
17:49 xiu left #gluster
17:56 mister_zombie1 joined #gluster
17:57 mister_zombie1 Hi, I have a question: using gluster 3.2  on centos 6, is there any way to make replicas be made on other nodes in preference?
17:58 mister_zombie1 I have 4 bricks in my volume
17:58 mister_zombie1 4 bricks 2 nodes.
17:59 JoeJulian First, why 3.2?
18:01 mister_zombie1 Corporative constraints.
18:02 manik joined #gluster
18:03 semiosis don't trust my ,,(ppa)?
18:03 glusterbot The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY
18:03 JoeJulian centos...
18:03 mister_zombie1 The networking guys follow CentOS releases. They also are not particularly keen on installing software they don't know, so if I ask them to install from sources they don't trust as well, they will laugh me off or just let the ticket grow mold before they treat it.
18:03 JoeJulian @yum repo
18:03 glusterbot JoeJulian: kkeithley's fedorapeople.org yum repository has 32- and 64-bit glusterfs 3.3 packages for RHEL/Fedora/Centos distributions: http://goo.gl/EyoCw
18:04 mister_zombie1 EPEL 6 is the best I can do.
18:04 JoeJulian Btw, the yum repo is from the guy who packages for epel
18:04 mister_zombie1 That's something I'll note in my list of things I need to tell them, thanks.
18:04 * semiosis doesn't trust your shady fly-by-night repo
18:04 semiosis j/k ;)
18:05 JoeJulian I guess. I have no tolerance for ignorance.
18:05 JoeJulian At least not today.
18:05 JoeJulian Anyway.. to answer the actual question:
18:05 JoeJulian @brick order
18:05 glusterbot JoeJulian: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
18:07 mister_zombie1 Sweet, thanks. I'll copy and paste that repo information as well, if I can make them yield on that, it will be a great day for my dev team.
18:08 joehoyle joined #gluster
18:09 coredumb joined #gluster
18:09 coredumb Hello
18:09 glusterbot coredumb: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
18:10 coredumb is there any performance advantage using NFS over the native client?
18:11 JoeJulian Depends on your definition.
18:11 coredumb definition of ?
18:11 coredumb :)
18:12 JoeJulian What problem are you trying to solve?
18:12 coredumb JoeJulian: no problem actually
18:12 coredumb just wondering what are the advantages to use NFS over the native clietn
18:13 JoeJulian There are some systems that can't use fuse.
18:13 JoeJulian nfs loses redundancy and some consistency.
18:13 coredumb that's what i thought
18:13 coredumb redundancy*
18:14 JoeJulian consistency due to kernel caching.
18:14 Supermathie JoeJulian: unless you're doing DNFS for multipathed NFS or keepalived or something to expose a failover IP for NFS... yeah native client is better though
18:15 JoeJulian Yeah, slighly incomplete answer, but I'm feeling really tired this morning for some reason.
18:16 Supermathie So many caveats to put in can get tiring :)
18:17 coredumb thanks i get the point :)
18:18 coredumb anyway anyone has tried using glusterfs over ZFS filesystems ?
18:18 mjrosenb [2013-04-11 14:17:33.706880] W [dht-selfheal.c:875:dht_selfheal_directory] 0-magluster-dht: 1 subvolumes down -- not fixing
18:18 mjrosenb [2013-04-11 14:17:37.681574] I [client.c:2090:client_rpc_notify] 0-magluster-client-1: disconnected
18:18 mjrosenb so I assume that this is in fact not normal.
18:19 JoeJulian Basically it's saying that it's not connected to one of the bricks.
18:22 mjrosenb glusterd  2975 2978     root    7u     IPv6              12455       0t0      TCP *:24007 (LISTEN)
18:22 mjrosenb it looks like both machines are listening.
18:24 JoeJulian @processes
18:24 glusterbot JoeJulian: the GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/hJBvL for more information.
18:25 JoeJulian So you're missing a glusterfsd. Should be able to check with "gluster volume status" (v3.3)
18:27 mjrosenb Brick memorybeta:/local24009NN/A
18:27 mjrosenb indeed.
18:28 wN joined #gluster
18:28 semiosis JoeJulian: nfs client option 'noac' (no attribute caching) helps get back that consistency with nfs clients, it's usually recommended (even in plain old nfs) when multiple writers are involved
18:28 semiosis s/JoeJulian/coredumb/
18:28 glusterbot What semiosis meant to say was: coredumb: nfs client option 'noac' (no attribute caching) helps get back that consistency with nfs clients, it's usually recommended (even in plain old nfs) when multiple writers are involved
18:28 coredumb semiosis: :)
18:28 coredumb i usually remove this
18:28 mjrosenb [2013-04-11 10:30:35.388320] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
18:29 coredumb yeah put "noac"
18:29 mjrosenb looks like i've been getting those every few seconds on that brick
18:30 H__ mjrosenb: i get these when i disable NFS
18:31 mjrosenb can I just kill glusterd if I want it to go away?
18:31 H__ yes
18:31 H__ it won't take glusterfsd's etc. with it
18:32 mjrosenb well, glusterfsd isn't running, so that isn't a concern
18:36 Supermathie Hey, so how do you tell glusterd to try and restart a glusterfsd that's died?
18:36 JoeJulian volume start $vol force
18:36 semiosis Supermathie: gluster volume start <volname> force
18:36 semiosis or restart glusterd, and it will spawn any missing glusterfsds when it starts
18:37 Supermathie semiosis: I can't remember what I tried, but I encountered this when suddenly a drive went flooey
18:37 Supermathie semiosis: Don't remember if I tried 'force', but I'll keep it in mind, thanks
18:38 semiosis yw
18:47 mjrosenb it would appear as if I have some init scripts to poke.
18:48 mjrosenb also, oh man, managing gluster these days is so much easier than it was ~4 years ago.
19:10 roo9 anyone happen to know where the bug report from http://www.gluster.org/2012/08/glus​terfs-bit-by-ext4-structure-change/ went? the review.gluster.com bug that's linked here doesn't work anymore
19:10 glusterbot <http://goo.gl/86YVi> (at www.gluster.org)
19:10 andreask joined #gluster
19:10 roo9 JoeJulian: I think that might be the cause of the issue you were helping me with the other day, the symptoms line up 10)%
19:10 roo9 *100%
19:11 JoeJulian @ext4
19:11 glusterbot JoeJulian: Read about the ext4 problem at http://goo.gl/PEBQU
19:11 JoeJulian Hmm
19:12 mister_zombie1 Say I create a volume with no replication and 2 bricks, if I add another node with 2 bricks as well, is there a way to enable replication after the facts? Could I change the order of the bricks as earlier discussed so that the replicas would live on the new node?
19:13 roo9 joejulian: I've read that post before and you wrote it and neither of us made that connection :) I really thought it was fixed though
19:13 JoeJulian bug 838784
19:13 glusterbot Bug http://goo.gl/CO1VZ high, high, ---, sgowda, MODIFIED , DHT: readdirp goes into a infinite loop with ext4
19:15 roo9 aha, thanks
19:16 JoeJulian @learn ext4 as Track the ext4 bugzilla report at http://goo.gl/CO1VZ
19:16 glusterbot JoeJulian: The operation succeeded.
19:20 kkeithley| http://review.gluster.org/#/c/4711/ and http://review.gluster.org/#/c/4799 (master/HEAD and release-3.4 respectively) have both been merged.
19:20 glusterbot Title: Gerrit Code Review (at review.gluster.org)
19:28 Supermathie So... let's say I wanted to convert ext4 to xfs. If I offline glusterd/glusterfsd on one of the nodes, reformat the bricks, restart glusterd, is it going to be smart and say "ERMAGERD!! WHERE'D MY FILES GO???" and replicate everything back?
19:28 jbrooks joined #gluster
19:29 bruce-s joined #gluster
19:33 semiosis Supermathie: yep, but you would be remiss if you didn't try that on a test cluster before doing it on prod
19:33 Supermathie I only have test :)
19:33 semiosis great
19:33 bruce-s I have 11TB in a LV which has been a geo-replication target and I want to turn it into a Gluster volume (still geo-rep target) so I can samba mount remotely to let client look at copy. What is most straightforward path to doing this?
19:33 Supermathie But the theory is sound?
19:34 semiosis also to compartmentalize any problems that may occur, i would suggest doing it one brick at a time, rather than all bricks on a server at a time, if you have more than one brick per server
19:34 semiosis the theory is sound
19:34 Supermathie BOOM TIME
19:36 JoeJulian bruce-s: stop geo-rep. Make your volume. geo-rep to the new volume. mount that new volume on your samba server.. ??? ... profit!
19:36 Supermathie bruce-s: errrr.... doesn't that LV need to have a filesystem on it to be a georep target? Can't you just run samba and share out that filesystem?
19:37 bruce-s currently it has xfs and is just a directory target of geo-rep
19:38 Supermathie bruce-s: So you can't just share that xfs filesystem with samba?
19:38 dustint joined #gluster
19:39 bruce-s I really want to incorporate stored replicated (remote) files in the new GV and haven't seen how to do that.
19:42 _pol_ joined #gluster
19:51 bruce-s joined #gluster
19:55 iatecake joined #gluster
19:57 Supermathie semiosis: Nice, it all worked. Had to start a heal though.
19:57 bruce-s is there a right way to take existing directory full of files and just make it a new brick in new gv?
19:57 iatecake joined #gluster
19:58 Supermathie Except this is weird.... I now have a strange date problem.
20:05 rwheeler joined #gluster
20:05 Supermathie Prior to doing the madness above I rsynced everything to a different gluster volume. Now the times between source and master are off by an hour.
20:09 semiosis Supermathie: great.  odd, but great
20:09 semiosis Supermathie: istr something about dates not being healed correctly :/
20:09 semiosis i'll try to find you the bug id
20:19 semiosis Supermathie: i think Bug 918437 is the one i was thinking ov
20:19 glusterbot Bug http://goo.gl/1QRyw urgent, unspecified, ---, pkarampu, MODIFIED , timestamps updated after self-heal following primary brick rebuild
20:19 semiosis s/ov/of/
20:19 glusterbot What semiosis meant to say was: Supermathie: i think Bug 918437 is the one i was thinking of
20:20 semiosis oh look, it's coffee-o-clock
20:23 Supermathie Yeah that looks like it.
20:24 mjrosenb does gluster support inotify on clients? (my guess is very 'no')
20:27 _pol joined #gluster
20:27 _pol joined #gluster
20:31 Supermathie mjrosenb: Nope
20:33 dustint joined #gluster
20:42 zaitcev joined #gluster
20:47 m0zes joined #gluster
21:06 Shdwdrgn joined #gluster
21:15 H__ mjrosenb: and i think it never will as inotify does not scale
21:19 Nagilum_ 0mq
21:20 mjrosenb H__: thus the 'very no'
21:45 joehoyle joined #gluster
21:50 fidevo joined #gluster
22:11 sefz joined #gluster
22:16 BSTR joined #gluster
22:32 _pol_ joined #gluster
22:33 _pol joined #gluster
22:56 jbrooks joined #gluster
23:05 H__ joined #gluster
23:15 Unidentified4773 joined #gluster
23:18 _pol joined #gluster
23:19 _pol joined #gluster
23:35 dbruhn left #gluster
23:46 eedfwchris joined #gluster
23:46 eedfwchris How can I I easily change the brick location on the same node (just changing mount points)
23:47 eedfwchris literally moving from like /mnt/b0 to /var/lib/glusterd/bricksB0
23:51 JoeJulian eedfwchris: pretty easy. Kill glusterfsd for the brick you want to move. Remount it where you want it (though /var/lib is technically the wrong place accoring to FHS), then use" gluster volume replace-brick myvol1 server:/mnt/b0 server:/data/glusterfs/myvol1/bricksB0 commit force"
23:54 _pol_ joined #gluster
23:55 _pol joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary