Camelia, the Perl 6 bug

IRC log for #gluster, 2013-10-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:02 bogen1 alright, so I removed a brick-pair (via remove-brick) and the data was not moved onto the other bricks first
00:03 bogen1 when I added the brick-pair back I had to rebalance to get the files back
00:03 bogen1 using glusterfs 3.2.7
00:04 bogen1 The problem is I'll be creating a several volumes across several servers
00:05 bogen1 then 1/2 servers will be moved to a different location (yeah, the timing is bad, but I have to have the volumes in place before the move starts)
00:05 bogen1 then the other 1/2 will move later
00:06 bogen1 so I guess I could remove the bricks, create a temp volume with the removed bricks
00:06 bogen1 then rsync the dat from the temp volume to main volume
00:07 bogen1 then remove the files from the temp volume
00:07 bogen1 then delete the temp volume
00:11 bogen1 or I could go with 4 way, then just shut down half the servers and wipe the data. I'll figure something out.
00:14 bogen1 I was just hoping there some built in way to say "I'm going to remove these bricks, stop writing to them and copy the data off to other bricks"
00:15 mjrosenb [2013-09-17 01:17:30.526729] W [fuse-bridge.c:1516:fuse_rename_cbk] 0-glusterfs-fuse: 458548: /incoming/sorted/a/redacted -> /incoming/sorted/b/redacted => -1 (Invalid cross-device link)
00:15 mjrosenb :-(
00:15 bogen1 well, when I rebalanced all the files seemed to come back, but rebalance status said the rebalance had failed
00:17 mjrosenb two questions about rebalancing: how do I initiate a rebalance?
00:18 mjrosenb and: if something goes wrong (brick crashes, glusterd goes insane), is there likely to be any unrecoverable damage?
00:18 bogen1 gluster volume rebalance VOLNAME start
00:18 bogen1 dunno
00:19 mjrosenb i'll wait for an answer to the second one before running that :-)
00:22 bogen1 yeah, I've just been using gluster for a few hours, so I'm not a reliable resource :)
00:24 bogen1 well, I need to leave work... I'll be back....
00:26 StarBeast joined #gluster
00:56 vpshastry joined #gluster
01:03 kPb_in joined #gluster
01:04 bsbrooks9 joined #gluster
01:10 mjrosenb ok, so as far as I can tell, there are two bricks both of which want all hashes for a particular directory.
01:10 mjrosenb but there are no errors saying this is happening.
01:10 mjrosenb which is confusing to say the least.
01:10 mjrosenb perhaps I should turn up the verbosity level?
01:13 bsbrooks99 joined #gluster
01:33 _BryanHm_ joined #gluster
01:40 bala joined #gluster
01:44 vynt joined #gluster
01:46 _ilbot joined #gluster
01:46 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:47 bharata-rao joined #gluster
01:51 harish joined #gluster
01:52 bala joined #gluster
01:58 mohankumar joined #gluster
02:00 bet_ joined #gluster
02:06 vpshastry joined #gluster
02:35 vpshastry joined #gluster
03:10 sgowda joined #gluster
03:15 bala joined #gluster
03:17 shubhendu joined #gluster
03:21 vpshastry joined #gluster
03:23 mohankumar joined #gluster
03:25 jag3773 joined #gluster
03:34 vpshastry left #gluster
03:58 kanagaraj joined #gluster
04:05 strager joined #gluster
04:05 dusmant joined #gluster
04:06 strager I created two Gluster nodes on two Vagrant (VirtualBox) VMs (running Ubuntu Precise 32-bit).  I made two bricks and a replicate volume (per the set-up docs) and both nodes seem set up properly.  When I touch a file in the brick of one VM, the file doesn't appear in the other brick.
04:07 strager I probably don't understand something about how GlusterFS works, but I can't find any documentation on how to ... do files.
04:07 strager I thought maybe I could mount the GlusterFS using NFS, but I can't find docs on how to actually do that.
04:08 rjoseph joined #gluster
04:08 itisravi joined #gluster
04:14 strager Ah, apparently there is a mount.glusterfs command which shows up nowhere in the 'getting started' guide.
04:15 Shdwdrgn joined #gluster
04:26 shylesh joined #gluster
04:29 bala joined #gluster
04:35 nueces joined #gluster
04:39 lalatenduM joined #gluster
04:41 ndarshan joined #gluster
04:41 bennyturns joined #gluster
04:42 raghu joined #gluster
04:42 kkeithley joined #gluster
04:44 vpshastry joined #gluster
04:46 portante joined #gluster
04:47 bfoster joined #gluster
04:47 _mattf joined #gluster
04:48 msvbhat joined #gluster
04:51 dblack joined #gluster
04:52 bdperkin joined #gluster
04:52 bala joined #gluster
04:59 shruti joined #gluster
05:03 ppai joined #gluster
05:07 aravindavk joined #gluster
05:13 CheRi joined #gluster
05:30 anands joined #gluster
05:37 bala joined #gluster
05:49 hagarth joined #gluster
05:58 kPb_in_ joined #gluster
06:00 glusterbot joined #gluster
06:04 glusterbot joined #gluster
06:05 glusterbot joined #gluster
06:05 davinder joined #gluster
06:07 ThatGraemeGuy joined #gluster
06:07 satheesh joined #gluster
06:23 ricky-ticky joined #gluster
06:23 hagarth joined #gluster
06:23 dusmant joined #gluster
06:24 ajha joined #gluster
06:26 bala joined #gluster
06:32 jtux joined #gluster
06:33 vimal joined #gluster
06:39 jag3773 joined #gluster
06:41 ricky-ticky joined #gluster
06:46 ctria joined #gluster
06:54 phil_ joined #gluster
07:02 Maxence joined #gluster
07:02 ekuric joined #gluster
07:02 meghanam joined #gluster
07:02 meghanam_ joined #gluster
07:02 ababu joined #gluster
07:03 blook joined #gluster
07:08 eseyman joined #gluster
07:20 Dga joined #gluster
07:35 hagarth joined #gluster
07:42 ThatGraemeGuy joined #gluster
07:44 ThatGraemeGuy_ joined #gluster
07:47 dusmant joined #gluster
07:51 ngoswami joined #gluster
07:56 bstr joined #gluster
07:57 andreask joined #gluster
08:16 psharma joined #gluster
08:31 StarBeast joined #gluster
08:41 spandit joined #gluster
08:46 raghu joined #gluster
08:56 rgustafs joined #gluster
09:01 sgowda joined #gluster
09:02 manik joined #gluster
09:04 ccha2 hello I have these errors
09:04 ccha2 E [socket.c:1715:socket_connect_finish] 0-NLM-client: connection to  failed (Connection timed out)
09:05 kshlm joined #gluster
09:10 giannello joined #gluster
09:10 satheesh joined #gluster
09:11 RameshN joined #gluster
09:26 shyam joined #gluster
09:29 glusterbot joined #gluster
09:29 ngoswami joined #gluster
09:31 harish joined #gluster
09:33 ccha2 every 30sec there are these messages
09:34 anands joined #gluster
09:36 ctria joined #gluster
09:38 eseyman joined #gluster
09:45 StarBeast joined #gluster
09:46 pkoro joined #gluster
09:49 rjoseph joined #gluster
09:50 JoeJulian ccha2: I've had that occasionally with versions prior to 3.4.1 (haven't yet with this version) and I could only cure it by remounting the client.
10:00 hagarth joined #gluster
10:01 ccha2 ok thank JoeJulian I will try it later
10:01 ccha2 using 3.3.2
10:01 ccha2 and nfs
10:01 ababu joined #gluster
10:04 itisravi joined #gluster
10:05 ccha2 Temporary, is there any bad thing about using mount glusterfs on some clients and mount nfs on other clients ?
10:16 satheesh joined #gluster
10:17 dusmant joined #gluster
10:18 mgebbe joined #gluster
10:19 shubhendu joined #gluster
10:20 RameshN joined #gluster
10:21 ndarshan joined #gluster
10:25 kanagaraj joined #gluster
10:27 aravindavk joined #gluster
10:27 ababu joined #gluster
10:33 andreask joined #gluster
10:44 F^nor joined #gluster
10:47 DV__ joined #gluster
10:53 ctria joined #gluster
11:11 jtux joined #gluster
11:14 aravindavk joined #gluster
11:21 ricky-ticky joined #gluster
11:22 CheRi joined #gluster
11:25 ppai joined #gluster
11:27 anands joined #gluster
11:37 tjikkun_work joined #gluster
11:40 bdperkin joined #gluster
11:42 ccha2 does 3.3.2 support nfs lock ?
11:45 kkeithley ccha2: yes
11:51 ccha2 hum
11:51 ccha2 I got error when I put flock on a file
11:51 ccha2 with nfs client
11:52 ccha2 and no error with glusterfs client
11:52 edward2 joined #gluster
11:58 Staples84 joined #gluster
11:59 CheRi joined #gluster
12:02 ndarshan joined #gluster
12:06 Alpinist joined #gluster
12:06 DV__ joined #gluster
12:09 kkeithley not flock() locks; POSIX advisory locks.  fcntl (fd, F_SETLK, ...)
12:11 dneary joined #gluster
12:11 kkeithley flock(2) doesn't lock on NFS, see the fine man page
12:16 tziOm joined #gluster
12:16 fuzzy_id joined #gluster
12:17 fuzzy_id is it right that gluster doesn't support ipv6?
12:24 _BryanHm_ joined #gluster
12:25 ccha2 kkeithley: it was php which use php flock()
12:28 mistich1 left #gluster
12:29 kkeithley unless you can configure php's flock to use fcntl(2) locking then you're out of luck. And FWIW, gluster tends not to be so good for php. php does a stat(2) on every include/import and that's just plain bad for gluster.  IIRC jdarcy's Negative Lookup xlator makes things a bit better for php.
12:30 ndevos @php
12:31 glusterbot ndevos: --fopen-keep-cache
12:31 glusterbot ndevos: (#1) php calls the stat() system call for every include. This triggers a self-heal check which makes most php software slow as they include hundreds of small files. See http://goo.gl/uDFgg for details., or (#2) It could also be worth mounting fuse with glusterfs --attribute-timeout=HIGH --entry-timeout=HIGH --negative-timeout=HIGH
12:31 kkeithley ah, thanks ndevos
12:40 bet_ joined #gluster
12:41 onny1 joined #gluster
13:03 fuzzy_id` joined #gluster
13:04 StarBeast joined #gluster
13:06 shyam joined #gluster
13:16 vynt joined #gluster
13:19 shubhendu joined #gluster
13:25 DV__ joined #gluster
13:30 jvyas joined #gluster
13:35 thiagol joined #gluster
13:36 jcsp joined #gluster
13:38 RameshN joined #gluster
13:38 ccha2 kkeithley: "IIRC jdarcy's Negative Lookup xlator makes things a bit better for php." <-- does this xlator is on an glusterfs version ?
13:38 cekstam joined #gluster
13:40 cekstam Hi! I have a strange problem where when I do a gluster peer probe on one peer after a complete gluster reset (rm -rf /var/lib/glusterfs/*) the other peer gets the IP address of the probing peers hostname as the peer? DNS resolution is working fine on both peers.
13:40 kkeithley no, and that was a red herring. See the glusterbot/ndevos comments above, they're better advice for using php.
13:41 daMaestro joined #gluster
13:41 kanagaraj joined #gluster
13:42 onny1 cekstam: I noticed this too. In this case one could just prefer ip addresses instead of hostnames ...
13:42 shyam joined #gluster
13:43 cekstam onny1 what did you do to resolve it? stopped glusterd and edited the peer file directly?
13:43 onny1 Don't know how to fix this. Maybe we can manually edit the peer list in /etc/glusterfs or /var/lib/glusterfs
13:43 onny1 Well I could give it a try :)
13:44 cekstam I'm running 3.4.0 right now, I did have a look at the release notes for 3.4.1, but I did not spot anything related to this.
13:46 hagarth joined #gluster
13:46 kkeithley If I'm not mistaken, you can probe "one peer" from "other peer" by hostname  and it will replace the IP address with "one peer's" hostname
13:47 kkeithley I don't recommend editing the files, but if you do, make sure you shut down all the gluster processes first, otherwise they may over-write your manual changes.
13:48 cekstam kkeithley, that did it. Thank you very much
13:49 onny1 yep that worked, thanks kkeithley
13:53 onny1 cekstam: well, you also have to change the ip address to hostname in the bricks/volumes
13:54 sgowda left #gluster
13:55 mattf joined #gluster
13:56 cekstam onny1 yes, have none yet, so I think I'll be fine. Thanks for the heads up though.
14:01 kaptk2 joined #gluster
14:01 shyam joined #gluster
14:13 bugs_ joined #gluster
14:15 mohankumar joined #gluster
14:15 DV__ joined #gluster
14:18 wushudoin joined #gluster
14:19 Excolo joined #gluster
14:22 Excolo Hi, I have a question that I cant seem to find an answer for, so was hoping someone in here could help me out. I have two servers setup with a basic replication setup. They lost connection between them for about a minute the other day, which caused glusterfsd to ramp up to 398% cpu usage according to running top on that box. Thing is, self heal is disabled on the entire volume (for a couple of reasons)... so what is it doing? why did i
14:22 Excolo t max out the cpu for a full day on that box, and is there any way to prevent it in the future (not necissarily what its doing, but just not have it max out the cpu so that it is still usable, as in this state it was too slow for our production system)
14:22 ndevos Excolo: maybe you use ,,(ext4)?
14:22 glusterbot Excolo: Read about the ext4 problem at http://goo.gl/Jytba or follow the bug report here http://goo.gl/CO1VZ
14:23 primusinterpares joined #gluster
14:23 Excolo I do, thank you. I didnt see that problem anywhere (maybe I was googling the wrong things? I was looking for anything pertaining to high cpu load and glusterfsd)
14:27 vpshastry joined #gluster
14:28 Excolo Thanks, this looks like it could be it. Been a pain in my ass for 3 months
14:29 vpshastry left #gluster
14:29 glusterbot New news from newglusterbugs: [Bug 1019874] Glusterfs crash when remove brick while data are still written <http://goo.gl/CJPThD>
14:30 zaitcev joined #gluster
14:30 failshell joined #gluster
14:31 ccha2 I have a problem with geo-replication
14:31 ccha2 I got this message everything minutes
14:31 ccha2 [2013-10-16 14:14:02.844728] E [posix-helpers.c:721:posix_handle_pair] 0-DATA-posix: /test/.svn/lock: key:trusted.glusterfs.76053997-b​cb1-4bee-8e8d-96a7aa74cf48.xtime error:No such file or directory
14:32 ccha2 this file doesn't exist, that was deleted since few days
14:32 ccha2 it looks replication want to get this file
14:33 ccha2 I stop replication, turn off index and restart replication
14:36 ccha2 how can clean it up ?
14:37 hagarth joined #gluster
14:43 ndk joined #gluster
14:46 phox "clean it up" and glusterfs... yeah, I'm working through an ugly kludge to get gluster to clean up another mess.
14:46 phox gluster is not designed for elegant recover from unanticipated situations.
14:48 jcsp joined #gluster
14:49 elyograg joined #gluster
14:49 elyograg I finally got new hardware to expand my gluster install.  I have questions ... queries ... posers.
14:51 satheesh1 joined #gluster
14:53 elyograg my existing setup is running 3.3.1.  If I just go forward with installing, the new machines will get 3.4 ... will that work? should I take steps to upgrade the existing machines to 3.4 first?
14:54 elyograg or should I put 3.3.1 on the new machines and then worry about upgrading?
14:57 sjoeboo joined #gluster
14:57 jag3773 joined #gluster
14:57 hagarth joined #gluster
14:58 ndk` joined #gluster
15:01 ndk joined #gluster
15:05 ctria joined #gluster
15:06 ctrianta joined #gluster
15:13 shylesh joined #gluster
15:13 bala joined #gluster
15:16 jbrooks joined #gluster
15:16 sprachgenerator joined #gluster
15:17 chirino joined #gluster
15:20 20WAAB4RY joined #gluster
15:20 shylesh joined #gluster
15:22 Technicool joined #gluster
15:23 jvyas elyograg, thats a good question : how does backwards compatibiliity with upgrading gluster work.  anyone know?
15:30 semiosis jvyas: sometimes it works, sometimes it doesnt.  notably 3.3 is not backwards compatible with 3.2, but 3.4 is (supposed to be) compatible with 3.3.  upgrade servers first, and see relevant ,,(upgrade) notes
15:30 glusterbot jvyas: I do not know about 'upgrade', but I do know about these similar topics: '3.3 upgrade notes', '3.4 upgrade notes'
15:31 jvyas well i guess one thing you could do:  back the whole directory up to some cheap storage , upgrade, create new volumes , and then CP the directorys into your mount.
15:32 ctria joined #gluster
15:32 semiosis thats always an option
15:32 zerick joined #gluster
15:34 ndk joined #gluster
15:39 badone joined #gluster
15:41 elyograg backing up is not really an option.  Already got 20TB on it.
15:41 semiosis s/always/sometimes/
15:41 glusterbot What semiosis meant to say was: thats sometimes an option
15:48 elyograg I saw the upgrade instructions, looks doable.  The goal is to get on the newer version everywhere.  I see three options right now: 1) Find the rpms (CentOS 6) for the older version and install the new machines with the same version I've got, upgrade later.  2) Install 3.4 on the new machines, upgrade the rest later.  3) Upgrade to 3.4, then install new machines.
15:48 aliguori joined #gluster
15:49 elyograg I've got four peers currently.  Two have bricks.  The other two are acting as network gateways, running a pacemaker cluster to provide a shared IP address, but they have no bricks.
15:50 elyograg most access is via NFS, there's a little bit via Samba.
15:50 hagarth joined #gluster
15:50 dusmant joined #gluster
15:51 SpeeR joined #gluster
16:02 Maxence_ joined #gluster
16:02 Maxence__ joined #gluster
16:02 hagarth joined #gluster
16:06 hagarth1 joined #gluster
16:09 jcsp joined #gluster
16:27 hagarth joined #gluster
16:30 Mo_ joined #gluster
16:31 hagarth1 joined #gluster
16:47 dusmant joined #gluster
16:47 mjrosenb so, if two bricks seem to have the same range specified for a single directory in dht, would I need to put a client into verbose mode for there to be errors in the log?
16:47 mjrosenb also how on earth do I fix this?
16:57 jclift_ joined #gluster
16:58 hagarth joined #gluster
17:05 andreask joined #gluster
17:09 rotbeard joined #gluster
17:15 MrNaviPacho joined #gluster
17:17 andreask joined #gluster
17:39 DV__ joined #gluster
17:41 lmickh joined #gluster
17:52 semiosis johnmark: ping
17:53 johnmark semiosis: yup
17:53 johnmark I saw your note :)
18:14 mjrosenb how does each brick decide on the value of trusted.glusterfs.dht?
18:19 lalatenduM joined #gluster
18:20 StarBeast joined #gluster
18:25 LoudNois_ joined #gluster
18:32 glusterbot New news from resolvedglusterbugs: [Bug 961856] [FEAT] Add Glupy, a python bindings meta xlator, to GlusterFS project <http://goo.gl/yCNTu>
19:00 failshel_ joined #gluster
19:14 giannello joined #gluster
19:24 davinder joined #gluster
19:31 glusterbot New news from newglusterbugs: [Bug 1020012] debug-trace.py in GlusterFS git no longer working <http://goo.gl/yJOeyO> || [Bug 892808] [FEAT] Bring subdirectory mount option with native client <http://goo.gl/wpcU0>
19:36 JoeJulian mjrosenb: There's a special xattr you can set through the client mount... let me find it again...
19:40 JoeJulian @fix-layout
19:40 glusterbot JoeJulian: I do not know about 'fix-layout', but I do know about these similar topics: 'targeted fix-layout'
19:41 JoeJulian @targeted fix-layout
19:41 glusterbot JoeJulian: You can trigger a fix-layout for a single directory by setting the extended attribute \"trusted.distribute.fix.layout\" to any value for that directory. This is done through a fuse client mount.
19:41 JoeJulian Heh, I had forgotten that I had already added that.
19:41 JoeJulian mjrosenb: ^^^
19:47 jclift joined #gluster
19:53 Alpinist joined #gluster
20:13 F^nor joined #gluster
20:14 ctria joined #gluster
20:14 giannello joined #gluster
20:15 hagarth joined #gluster
20:20 voronaam joined #gluster
20:25 purpleidea joined #gluster
20:31 pdrakeweb joined #gluster
20:34 voronaam Hi, today we had a power outage on a small 2-server cluster. Once power was restored, one of the servers had issues and was not up immediately. The gluster on the second tried to heal itself, but could not do it without another replica. So it started up, but NFS server was not available. Everything was back to normal once the other server was repaired. The question: what should've I done if I lost the second server for good? Is there a command to tell
20:44 jbrooks joined #gluster
20:50 badone joined #gluster
21:03 JoeJulian voronaam: nfs is served from all servers, but since there's no redundancy in the tcp connection, most people work around that by using a floating ip and some sort of control software to float that ip.
21:05 voronaam Thanks. That's not exactly what happened. The problem was when out of two servers (replica factor = 2) only one was available after reboot. That single server did not start its NFS server until the other server booted up
21:05 JoeJulian voronaam: that's atypical. I would check the nfs log to see why that happened.
21:06 voronaam ok. let me put snippets from the logs to pastebin
21:08 DV__ joined #gluster
21:09 bogen1 joined #gluster
21:09 bogen1 RuntimeError: not a valid option: gluster-command-dir"
21:10 bogen1 when updating from gluster 3.2.7 to 3.4.0
21:12 sprachgenerator joined #gluster
21:13 voronaam http://pastebin.com/pAstMKYK
21:13 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
21:13 voronaam Oh, sorry
21:13 voronaam http://fpaste.org/47309/38195802/
21:13 glusterbot Title: #47309 Fedora Project Pastebin (at fpaste.org)
21:16 voronaam JoeJulian, that is not a big issue - this cluster is considered experimental and is not in 24/7 production mode. I just wanted to know what to do in case one of the servers is broken after power outage (that can happen again)
21:16 JoeJulian Heh, url shortener fail... "Shortened URL: #ur1.ca is down"
21:17 JoeJulian voronaam: None of those is the nfs log though. That's what /should/ show what happened with nfs.
21:19 voronaam Actually, nfs.log only has some of the lines for the other log file. But anyway, here it is: http://fpaste.org/47311/95833713/
21:19 glusterbot Title: #47311 Fedora Project Pastebin (at fpaste.org)
21:19 voronaam That's from /var/log/glusterfs/nfs.log
21:22 JoeJulian How about volume info
21:23 voronaam http://fpaste.org/47314/38195861/
21:23 glusterbot Title: #47314 Fedora Project Pastebin (at fpaste.org)
21:24 voronaam All the logs are from the 10.19.8.85 sever. Server 10.19.8.104 was down
21:28 JoeJulian 10.19.8.35 must be a client?
21:28 voronaam Yes
21:31 JoeJulian voronaam: Is this 3.4.0 or 3.4.1?
21:31 JoeJulian Ah, 3.4.0
21:31 JoeJulian That's why.
21:32 JoeJulian Upgrade to 3.4.1. There's a known bug when glusterd blocking when starting services.
21:32 voronaam Great! Thanks!
21:38 Alpinist joined #gluster
21:54 fidevo joined #gluster
22:02 StarBeas_ joined #gluster
22:14 masterzen joined #gluster
22:49 khushildep joined #gluster
23:07 druonysuse joined #gluster
23:39 StarBeast joined #gluster
23:59 mjrosenb JoeJulian: awesome, thanks!

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary