Camelia, the Perl 6 bug

IRC log for #gluster, 2013-07-17

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 tqrst hrm. 2 of my servers have become separated from their peers: http://fpaste.org/25826/40187921/. "Unable to find hostname: some_ip_that_is_actually_j​ust_fine_thankyouverymuch"
00:00 glusterbot Title: #25826 Fedora Project Pastebin (at fpaste.org)
00:00 tqrst the other servers are working fine as far as I can tell
00:07 yinyin joined #gluster
00:17 lpabon joined #gluster
00:19 JoeJulian tqrst: Most common reason I've seen for peers to be rejected is because a volume definition was out of sync. Just rsync /var/lib/glusterd/vols from a known good peer and restart glusterd.
00:24 tqrst JoeJulian: doing that atm. This was tingling my "I've seen this before sense", which brought me back to http://irclog.perlgeek.de/gluster/2012-11-26. That was when the volume sync command would fail and spit out non-ascii gibberish. Good times.
00:24 puebele joined #gluster
00:25 JoeJulian Yeah, that was awesome.
00:25 tqrst still happens, by the way
00:25 tqrst I'd forgotten about that
00:26 * JoeJulian raises an eyebrow...
00:27 xdexter T0aD, do you use rsync with files-from?
00:28 T0aD i use rdiff-backup
00:29 xdexter files-from looks good for i
00:30 tqrst JoeJulian: rsyncing, restarting and reprobing fixed the peer issues
00:31 JoeJulian cool. That's why we need jdarcy's suggestion of a managed configuration service.
00:34 joelwallis joined #gluster
00:34 tqrst there, now everything is [finally] under $mount/brick insted of $mount
00:35 tqrst instead*
00:35 JoeJulian nice
00:35 JoeJulian I haven't even gotten that completed myself.
00:36 tqrst figured I might as well do it after everything segfaulted :\
00:45 _pol joined #gluster
00:46 T0aD im gonna show you a sexy thing
00:47 _pol joined #gluster
00:53 T0aD http://www.bpaste.net/show/w1WKS4hOTRqRtEuZUkpd/
00:53 glusterbot <http://goo.gl/En01J> (at www.bpaste.net)
00:56 T0aD hmm of course disk usage aint updated.
00:56 bala joined #gluster
00:57 [o__o] joined #gluster
01:00 harish joined #gluster
01:02 tqrst just went from 3.3.1 to 3.4. First server up, and its logs are getting spammed with "[2013-07-17 01:02:03.069471] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)" :\
01:04 JoeJulian tqrst: You're doing a rolling upgrade then?
01:05 tqrst JoeJulian: no, everything's off right now
01:05 tqrst except for this one server
01:06 tqrst (and by spammed I mean ~6 every second)
01:06 JoeJulian yee-haw
01:06 tqrst yeah, fun
01:07 tqrst the timestamps look off, too
01:07 tqrst apparently it's 1am now?
01:07 JoeJulian gmt
01:08 tqrst oh, the output timestamp format changed then?
01:08 JoeJulian yes
01:08 JoeJulian Should have been iso, imho.
01:09 tqrst or whatever syslogd wants to tack on
01:09 tqrst maybe this should be in the release notes
01:11 JoeJulian No, not syslogd... unless you've configured syslogd to use the standard iso format. There's a reason the iso committee chose a standard. Let's use it.
01:11 tqrst there's a relevant xkcd for this, but I'd rather fix my volume first :p
01:11 JoeJulian hehe, I've seen it. :D
01:12 tqrst I found a related mailing list post but the author simply wiped his whole config by the looks of it
01:12 T0aD http://www.bpaste.net/show/vHuodArZqFWhayYpgprz/ oh yeah.
01:12 glusterbot <http://goo.gl/TFVWW> (at www.bpaste.net)
01:15 tqrst looks like the mount syntax changed, too
01:15 tqrst [2013-07-17 01:13:38.982720] E [mount.c:298:gf_fuse_mount] 0-glusterfs-fuse: mount of ml43:/bigdata to /mnt/bigdata (default_permissions,noatime,nodir​atime,allow_other,max_read=131072) failed
01:16 tqrst [2013-07-17 01:13:38.978227] I [mount.c:290:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Invalid argument), retry to mount via fusermount
01:17 tqrst wow, and I thought the transport endpoint log message spam from nfs was bad
01:17 T0aD http://www.bpaste.net/show/V36Kof2NXZFLfKteWDM1/ 10,000 is too much apparently.
01:17 glusterbot <http://goo.gl/qJn33> (at www.bpaste.net)
01:21 tqrst and "Server and Client lk-version numbers are not same", hrm
01:21 glusterbot tqrst: This is normal behavior and can safely be ignored.
01:21 tqrst ok, the readv spam stopped once all my servers were up
01:33 tqrst the main source of spam in the logs left is '[2013-07-17 01:32:11.479712] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused)' in glusterd, which happens every 3-4 seconds.
01:33 raghug joined #gluster
01:43 T0aD 2500 quotas work fine
01:43 harish joined #gluster
01:44 kevein joined #gluster
01:44 T0aD so far so good for 5000
01:46 T0aD 7500 boom!
01:59 bala joined #gluster
02:51 vshankar joined #gluster
02:56 mtrythall joined #gluster
02:57 mtrythall joined #gluster
03:01 kshlm joined #gluster
03:05 sprachgenerator joined #gluster
03:11 raghug joined #gluster
03:13 bharata-rao joined #gluster
03:26 kshlm joined #gluster
03:30 badone joined #gluster
03:33 ujjain joined #gluster
03:37 hagarth joined #gluster
03:39 sgowda joined #gluster
04:08 lmickh joined #gluster
04:11 lmickh question: do gluster bricks have to be tied directly to devices?  Is it bad practice to just export a folder that is on a drive for example?
04:25 hagarth joined #gluster
04:44 rastar joined #gluster
04:46 samppah_ lmickh: glusterfs is tied to directories so you must ensure that device is mounted first
04:49 lmickh samppah_: I got that.  Was just wondering if it matters if you export sub directories.  Ex: /dev/sdb1 is mounted to /export/sdb1, but I export /export/sdb1/stuff and /export/sdb1/other
04:51 rcoup joined #gluster
04:54 rjoseph joined #gluster
04:56 samppah_ lmickh: it should be fine and i use same approach myself :)
04:56 lmickh samppah_: great. thanks.
04:57 mohankumar joined #gluster
05:05 mtrythall joined #gluster
05:06 CheRi joined #gluster
05:24 bala joined #gluster
05:25 lalatenduM joined #gluster
05:29 shylesh joined #gluster
05:30 vpshastry joined #gluster
05:37 badone joined #gluster
05:41 ramkrsna joined #gluster
05:41 ramkrsna joined #gluster
05:48 vpshastry1 joined #gluster
05:54 bulde joined #gluster
05:58 glusterbot New news from resolvedglusterbugs: [Bug 964021] Lock migration should migrate lock state in server connection object <http://goo.gl/wBa7I>
05:59 deepakcs joined #gluster
06:03 rgustafs joined #gluster
06:10 guigui3 joined #gluster
06:15 raghu joined #gluster
06:16 psharma joined #gluster
06:18 vpshastry1 joined #gluster
06:20 roidelapluie joined #gluster
06:20 roidelapluie \o/
06:21 samppah_ o//
06:28 mooperd joined #gluster
06:29 Recruiter joined #gluster
06:33 raghu joined #gluster
06:42 jtux joined #gluster
06:43 ekuric joined #gluster
06:45 glusterbot New news from newglusterbugs: [Bug 949406] Rebalance fails on all the nodes when glusterd is down on one of the nodes in the cluster <http://goo.gl/Q8dyW>
06:46 ekuric joined #gluster
06:49 krink joined #gluster
06:52 ramkrsna joined #gluster
06:54 ctria joined #gluster
06:55 ricky-ticky joined #gluster
07:07 jtux joined #gluster
07:10 avati_ joined #gluster
07:19 hybrid512 joined #gluster
07:27 kshlm joined #gluster
07:27 ramkrsna joined #gluster
07:31 andreask joined #gluster
07:33 nightwalk joined #gluster
07:35 mooperd joined #gluster
08:05 badone joined #gluster
08:09 hybrid512 joined #gluster
08:16 glusterbot New news from newglusterbugs: [Bug 847842] [FEAT] Active-Active geo-replication <http://goo.gl/Z41og>
08:23 puebele1 joined #gluster
08:28 hagarth joined #gluster
08:43 skyw joined #gluster
08:44 coredumb am i the only to find that the .repo for yum repositories are incorrect? http://download.gluster.org/pub/gluster/glu​sterfs/3.4/3.4.0/CentOS/glusterfs-epel.repo $releasever is expanded in 6.x on an EL6 system
08:44 glusterbot <http://goo.gl/bxJ1h> (at download.gluster.org)
08:48 dpaul joined #gluster
08:50 mjrosenb joined #gluster
08:50 haidz joined #gluster
08:54 piotrektt joined #gluster
08:54 piotrektt joined #gluster
08:55 eryc joined #gluster
08:55 eryc joined #gluster
08:57 vimal joined #gluster
09:08 rcoup joined #gluster
09:15 spider_fingers joined #gluster
09:24 shylesh joined #gluster
09:25 hagarth joined #gluster
09:29 glusterbot New news from resolvedglusterbugs: [Bug 948643] gluster volume status --xml outputs wrong xml structure <http://goo.gl/0k527>
09:38 bala joined #gluster
10:11 shylesh joined #gluster
10:45 bulde1 joined #gluster
10:59 mynameisbruce joined #gluster
11:00 harish joined #gluster
11:04 vpshastry1 joined #gluster
11:04 edward1 joined #gluster
11:10 andreask joined #gluster
11:14 CheRi joined #gluster
11:30 21WAA7P12 joined #gluster
11:33 rcheleguini joined #gluster
11:33 bluefoxxx joined #gluster
11:33 bluefoxxx Can't find any docs on making glusterfs panic on error.
11:34 bluefoxxx I got an I/O error trying to access a mounted file system, which i remediated by reboot
11:34 bluefoxxx would like to set kernel.panic=5 and errors=panic on that
11:35 lpabon joined #gluster
11:39 puebele1 joined #gluster
11:49 glusterbot New news from newglusterbugs: [Bug 985390] Posix ACL not supported on debian os <http://goo.gl/Pp6sC>
11:52 kkeithley @later JoeJulian: adding the Before=network-online.target in glusterd.service works for me. The trick seems to be not using _netdev in your /etc/fstab. If I change defaults to _netdev then it doesn't work. (defaults,_netdev seems to work too)
12:03 yinyin joined #gluster
12:03 bluefoxxx watchdog then.
12:16 bulde joined #gluster
12:23 aliguori joined #gluster
12:25 jskinner_ joined #gluster
12:29 CheRi joined #gluster
12:31 rgustafs joined #gluster
12:33 _BuBU joined #gluster
12:33 _BuBU Hi
12:33 glusterbot _BuBU: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
12:34 _BuBU I've an issue adding brick
12:34 _BuBU ~> gluster volume add-brick download replica 2 172.20.8.222:/mnt/glusterfs/download
12:34 _BuBU Brick 172.20.8.222:/mnt/glusterfs/download is already a part of the volume
12:35 _BuBU and with gluster volume info I've
12:35 _BuBU Bricks:
12:35 _BuBU Brick1: 172.20.8.221:/mnt/glusterfs/download
12:35 _BuBU only
12:35 sprachgenerator joined #gluster
12:36 T0aD weird
12:36 ndevos _BuBU: sounds like /mnt/glusterfs/download on 172.20.8.222 has a xattr set that indociates it has been used bfore
12:36 T0aD attr -l /mnt/glusterfs/download ; ls -ld /mnt/glusterfs/download/.glusterfs
12:36 ndevos *before
12:37 T0aD yeah gluster should be definitely more verbose about it instead of letting us guess
12:38 _BuBU nothing for both attr and .glusterfs
12:38 _BuBU the 222 server is a newly installed server
12:38 ndevos _BuBU: try 'getfatt -m . -d -e hex /mnt/glusterfs/download ,8,10,11,12,13,14,15,16,17,18,19,20,21,​22,23,24,25,26,27,28,29,30,31,32,33,35]                              '
12:39 ndevos aah!
12:39 T0aD aaah !
12:39 T0aD you scared me!
12:39 ndevos getfatt -m . -d -e hex /mnt/glusterfs/download /mnt/glusterfs /mnt
12:39 T0aD getfattr -m . -d /mnt/glusterfs/download
12:39 ndevos getfatt+r indeed
12:39 T0aD who cares about -e hex
12:40 ndevos well, -e hex makes it easier to compare values
12:40 _BuBU - /mnt/glusterfs/download> getfattr -m . -d /mnt/glusterfs/download
12:40 _BuBU - /mnt/glusterfs/download>
12:40 skyw joined #gluster
12:40 T0aD its easy already in base64 or whatever that it
12:40 rwheeler joined #gluster
12:41 bennyturns joined #gluster
12:43 T0aD oh yeah maybe you re right
12:43 T0aD its not so complex in base64 but it already has more meaning in hex
12:44 T0aD trusted.glusterfs.quota.dirty=0x3000
12:44 T0aD ouh dirty girl
12:44 _BuBU strange things is:
12:44 _BuBU > gluster volume remove-brick download 172.20.8.222:/mnt/glusterfs/download
12:45 _BuBU Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
12:45 _BuBU Deleting all the bricks of the volume is not allowed
12:45 _BuBU whereas the one registered is for 221
12:45 T0aD it seems thats the only working brick of the volume
12:45 T0aD are 221 and 222 two different servers ? :P
12:45 _BuBU and I initially created the volume on 221
12:45 _BuBU yes 2 differents servers
12:46 _BuBU ~> ip addr list|grep 172.20.8.222
12:46 _BuBU ~> ip addr list|grep 172.20.8.221
12:46 _BuBU inet 172.20.8.221/24 brd 172.20.8.255 scope global bond0.8
12:47 _BuBU ~> gluster peer detach 172.20.8.222
12:47 _BuBU 172.20.8.222 is localhost
12:47 _BuBU when I'm on 221
12:47 T0aD it seems 222 is localhost on 221 then
12:48 _BuBU ~> gluster peer detach 172.20.8.221
12:48 _BuBU 172.20.8.221 is localhost
12:48 _BuBU doing this on the same box
12:48 ndevos _BuBU: could you check if /var/glusterd/glusterd.info contains a different UUID for each server?
12:49 _BuBU - /var/lib/glusterd> cat glusterd.info
12:49 _BuBU UUID=25a07f1a-a995-402f-a7c2-1a84197f84ff
12:49 _BuBU on the first server
12:49 glusterbot New news from newglusterbugs: [Bug 985406] Cannot change file permissions from windows client <http://goo.gl/kRe7w>
12:49 T0aD ndevos, whats the point ? the 2 ips are on the same machine
12:49 _BuBU - /var/lib/glusterd> cat glusterd.info
12:49 _BuBU UUID=25a07f1a-a995-402f-a7c2-1a84197f84ff
12:49 _BuBU on the second one
12:50 _BuBU ToaD: IPs are NOT on the same machines !
12:50 ndevos ~cloned servers | _BuBU
12:50 glusterbot _BuBU: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
12:51 _BuBU ndevos: indeed the 2 boxes are cloned
12:51 _BuBU so that's the issue
12:51 _BuBU thx for the hint
12:52 * ndevos earned himself an ice-cream
12:52 evilkitty joined #gluster
12:54 lalatenduM joined #gluster
12:55 T0aD _BuBU, ah sorry didnt see the separation clearly
12:55 T0aD ndevos, haha
12:55 evilkitty Hi, i keep running into the error "can't locate object method "new" via package" when trying to run a module in pocoirc. Can anyone help please?
12:56 T0aD evilkitty, sure, whats your problem ?
12:56 evilkitty im trying to run hailo in pocoirc and i keep getting that error
12:56 evilkitty i'm new... so i dont know what to do
12:56 evilkitty i have searched google though and i've got nothing
12:57 T0aD i see.. but whats the exact problem ?
12:57 _BuBU did someone already upgarded production from 3.3 to 3.4 ? and how is the migration/upgrade ?
12:57 evilkitty it says
12:57 evilkitty rror !!! Event _start in session 1 raised execption:
12:57 evilkitty error*
12:58 evilkitty can't locate object method "new" via package "Hailo" at
12:58 evilkitty and wont run
12:58 ndevos I dont think that has anything to do with gluster, evilkitty
12:59 evilkitty oh, im sorry
12:59 evilkitty bye
12:59 evilkitty left #gluster
12:59 T0aD some people
13:01 mohankumar joined #gluster
13:02 kkeithley ndevos earns himself an icecream++
13:02 kkeithley Maybe a Dame Blanche even
13:03 kkeithley Oh, you're in Netherlands, not Belgium. Never mind
13:13 vpshastry joined #gluster
13:13 plarsen joined #gluster
13:19 glusterbot New news from newglusterbugs: [Bug 985424] Gluster 3.4.0 RDMA stops working with more then a small handful of nodes <http://goo.gl/aUS6T>
13:20 yinyin joined #gluster
13:22 ctria joined #gluster
13:29 hagarth joined #gluster
13:33 jdarcy joined #gluster
13:41 _pol joined #gluster
13:45 puebele1 joined #gluster
13:45 joelwallis joined #gluster
13:49 ndevos lol, kkeithley, but its the italians that are famous for their icecream
13:50 bugs_ joined #gluster
13:52 kkeithley yes, gelato is good too
13:52 kkeithley very good
13:54 T0aD with their pooping ice cream ? :P
13:55 dewey joined #gluster
13:57 kaptk2 joined #gluster
14:02 lpabon joined #gluster
14:03 ctria joined #gluster
14:05 andreask joined #gluster
14:12 T0aD whats gsyncd.py for ?
14:14 yinyin joined #gluster
14:19 mtrythall joined #gluster
14:21 andreask joined #gluster
14:22 andreask joined #gluster
14:23 theron joined #gluster
14:23 andreask joined #gluster
14:28 tqrst coredumb: late reply, but yes I have the same issue here; I always end up having to hardcode $releasever to 6 :\
14:28 jag3773 joined #gluster
14:29 duerF joined #gluster
14:31 * tqrst eyes glusterfs's memory usage suspiciously
14:32 tqrst went from 1.5% to 7.1% after 12 hours of rebalancing on a box that has 16G
14:32 tqrst also, is it just me or is bugzilla.redhat.com horrendously slow on a good day?
14:34 T0aD couldnt say
14:34 coredumb tqrst: yeah i found it weird that no one reported it :D
14:34 tqrst coredumb: I figured it was a glitch specific to scientificlinux
14:34 tqrst but maybe not, then
14:37 kkeithley tqrst, coredumb: I used the glusterfs-epel.repo file on a centos 6.1 box without having to change it. I'll try a newer centos. What release of centos are you using?
14:37 tqrst kkeithley: 6.1
14:38 tqrst and I'm on scientificlinux, but it's basically just centos
14:38 kkeithley yes
14:40 coredumb tqrst: i'm also using SL here so maybe it only hits SL
14:40 coredumb i don't remember centos using 6Server and the likes
14:40 coredumb but i may be wrong
14:40 tqrst coredumb: which version are you on? I'm curious if you encountered any stability issues with gluster 3.3.1. We're on 6.1, and have had a terrible time with segfaults whenever we glance in gluster's general direction.
14:41 tqrst just calling replace-brick yesterday brought all gluster{d,fs,fsd} down
14:42 coredumb tqrst: i'm on 6.4 with 3.4
14:42 coredumb but i don't anything with my volumes :D
14:42 kkeithley I'll try SL 6.4 instead then
14:45 T0aD i guess gsyncd.py is used to replicate configuration files
14:49 tqrst looks like I still won't be able to rebalance with 3.4
14:49 tqrst memory usage is (slowly) going up, and showing no signs of stopping :\
14:50 T0aD is that an issue ?
14:51 tqrst I'm not sure if anyone's encountered the same problem. I posted about it on the mailing list a while back but got no replies.
14:51 semiosis T0aD: gsyncd is used for geo-replication.  config files (in /var/lib/glusterd) are managed (& replicated) by glusterd.
14:51 T0aD semiosis, oh thanks
14:51 semiosis yw
14:52 T0aD is there a way to manually trigger a replication of config files ?
14:52 semiosis gluster volume sync
14:52 T0aD really ?
14:52 T0aD and it will copy from local to others  peers then ?
14:52 semiosis although 9/10 times i'm about to use that command the configs get synced automatically before i have a chance to
14:53 T0aD someone should really document this
14:53 semiosis T0aD++
14:53 T0aD always wondered what it was, never saw it in manpage
14:53 semiosis go for it, thanks
14:53 T0aD you want me to document this ?
14:53 mtrythall joined #gluster
14:54 mtrythall left #gluster
14:54 T0aD yeah nothing in PDF nor manpage
14:54 bala joined #gluster
14:56 dbruhn joined #gluster
14:56 T0aD source doesnt seem to be documented as well
14:56 semiosis gluster volume help
14:56 T0aD gluster volume sync help
14:56 semiosis does that do something?
14:57 T0aD ah only for all the rest
14:57 T0aD like gluster volume quota help
14:57 T0aD maybe it just spits the usage of that line when the volume name aitn found
14:58 T0aD volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
14:58 T0aD so i guess if i were to use localhost as source it would be gluster volume sync localhost all
14:58 semiosis that doesnt make sense
14:59 T0aD sorry ?
14:59 semiosis sync from localhost to localhost should not do anything
14:59 semiosis since localhost === localhost
14:59 T0aD ah yeah my bad
14:59 T0aD its not to spread changes
14:59 T0aD its to copy remote changes locally
14:59 T0aD so it has to be done on every brick hm hm
14:59 tqrst every server, rather
15:00 T0aD yeah
15:00 semiosis T0aD: so you edited volfiles on one server & now you want those changes synced to the rest?
15:01 T0aD was looking for a way other than cp ./some_config_files ./dir; scp ./dir user@host:./; then log into that box copy the files in /var/lib/gluster
15:01 semiosis also be aware that (iirc) sync only works when the volume to be synced doesnt exist on the local server
15:01 T0aD how is that possible
15:01 T0aD jeez
15:01 tqrst also, removing vols is not enough for the server to think the volume doesn't exist
15:01 tqrst I always end up just scping
15:01 T0aD like when you just add a new peer ?
15:01 satheesh joined #gluster
15:01 semiosis T0aD: yes or replace a server & need to sync it up
15:02 T0aD well thats not good then
15:02 T0aD (in my specific case)
15:02 tqrst the only time I managed to get sync to do something, it segfaulted and output something about a volume whose name had non-ascii in it :\
15:02 semiosis considering that editing volfiles is explicitly discouraged, deprecated, unsupported, etc...
15:02 semiosis it's not surprising that it's hard
15:02 T0aD its not hard for glusterd
15:03 T0aD and thats the way the soft works, out of configuration files, and you can real reasons to be willing to edit them
15:03 semiosis ok
15:03 T0aD you can have real*
15:04 tqrst semiosis: since it's discouraged, how would you recommend updating certain parts of a configuration file that gluster complains about like listen-port now being transport.socket.listen-port?
15:04 semiosis sounds good in theory, but reasons alone won't get you very far in practice
15:04 T0aD semiosis, yeah sure whatever dude.
15:04 T0aD be happy with your nonsense
15:04 * semiosis :D
15:05 semiosis tqrst: gotta be gluster volume set
15:05 semiosis that's the recommended way to set config
15:05 semiosis also that sounds like an undocumented option so ymmv using it
15:05 semiosis or i'm not up to speed on the latest options
15:05 tqrst [2013-07-17 01:18:33.232295] W [options.c:848:xl_opt_validate] 0-bigdata-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
15:05 tqrst welp
15:06 raghug joined #gluster
15:06 tqrst unless the logs are recommending undocumented options...
15:06 semiosis interesting
15:06 kedmison joined #gluster
15:06 semiosis tqrst: did that appear after an upgrade from a previous gluster release?
15:06 tqrst semiosis: yes. 3.3.1 to 3.4
15:07 tqrst (everything segfaulted yesterday, so I used the downtime to update to 3.4 while I was at it)
15:07 semiosis tqrst: i suspect next time the files are generated & saved to disk that may be corrected, if not probably worth it to file a bug about it
15:07 glusterbot http://goo.gl/UUuCq
15:07 tqrst semiosis: yeah, I was expecting gluster to update the files by itself, but they're still the same even after restarting
15:08 semiosis maybe some config change (via gluster volume set) is needed to trigger that, not sure
15:08 T0aD semiosis, arent you going to tell him its not recommended not to use GA ?  :)
15:08 semiosis @custom config
15:08 tqrst these are brick-specific options, though, not volume-specific
15:09 semiosis tqrst: volume set commands can affect brick & client config files
15:09 tqrst ah, ok
15:09 semiosis brick-log-level for example
15:09 semiosis & others
15:10 semiosis 3.3.1 & 3.4.0 are both GA (although 3.3.2 is out now)
15:11 T0aD Latest Version: GlusterFS 3.4 (ver. 3.4.0 – July 14, 2013)
15:11 T0aD waow just recently
15:11 tqrst this update went much more smoothly than 3.2.whatever to 3.3
15:11 T0aD n/m my wise crack then :)
15:12 ramkrsna joined #gluster
15:18 T0aD http://www.bpaste.net/show/uQSJ7yQBMAMUD3AkjKuv/ haha
15:18 glusterbot <http://goo.gl/qEGwn> (at www.bpaste.net)
15:18 plarsen joined #gluster
15:28 risibusy joined #gluster
15:34 failshell joined #gluster
15:40 daMaestro joined #gluster
15:42 T0aD insteresting how gluster saves directory sizes in xattributes
15:42 lkoranda joined #gluster
15:43 nightwalk joined #gluster
15:43 tqrst what can gluster become split-brain *about* when we're talking about folders? Permissions, {m,c,a}time, anything else?
15:43 vpshastry1 joined #gluster
15:44 chirino joined #gluster
15:44 semiosis probably entries too
15:45 kedmison joined #gluster
15:50 krink joined #gluster
15:54 jebba joined #gluster
15:59 _pol joined #gluster
15:59 bulde joined #gluster
16:02 vpshastry1 left #gluster
16:03 yinyin joined #gluster
16:05 kedmison joined #gluster
16:17 dewey I have an experimental an unimportant gluster setup that I just upgraded to 3.4.  Now on one node when I run "gluster volume status" I get "Another transaction could be in progress. Please try again after sometime."  I would like to understand and repair this issue.  Failing that I would like to hit it on the head with a hammer and make it go away so I can get on with playing with gluster. ...
16:17 dewey ...Google fails me.  Any help appreciated.
16:18 T0aD its probably a lock on the manager
16:19 T0aD i got the same issue when issuing 1 command from one server while another server was busy answering one command (like gluster volume quota users list on 1st server, and another quota command on the 2nd server)
16:19 jclift_ joined #gluster
16:20 dewey Can you give me some clue as to how I might get past it?  It's been persistent since upgrade (~10 minutes).
16:20 dewey Other nodes report 1 sepecific brick of this node offline
16:21 T0aD no idea dude, just starting again on glusterfs
16:21 T0aD for me it was the end of the first command that allowed me to run commands on the second server
16:22 dewey OK thanks.  I'll go looking for a big hammer <grin/>
16:22 T0aD you might want to ps ax -H | grep gluster on all servers to see if one command aint running
16:22 T0aD :)
16:23 tqrst coughpgrepcough
16:23 T0aD https://github.com/gluster/glusterfs/pull/16
16:24 glusterbot Title: Quota new design by vpshastry · Pull Request #16 · gluster/glusterfs · GitHub (at github.com)
16:24 T0aD jeez i didnt see that, lets read before patching quota up
16:25 zaitcev joined #gluster
16:26 T0aD so far its setting soft / hard limits
16:27 guigui4 joined #gluster
16:29 raghug joined #gluster
16:30 tqrst "Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume."; sure, let me just rm -rf /myvolume, that sounds very sensible.
16:33 mafrac joined #gluster
16:33 T0aD jeez
16:34 mafrac Hello all. Is it possible upgrade to gluster 3.4 on centos 5.9?
16:34 T0aD there is no tool out there to help you see what is splitted ? thats what i will do in your case
16:34 semiosis ~split brain | tqrst
16:34 glusterbot tqrst: I do not know about 'split brain', but I do know about these similar topics: 'split-brain', 'splitbrain'
16:34 semiosis ~split-brain | tqrst
16:34 glusterbot tqrst: To heal split-brain in 3.3, see http://goo.gl/FPFUX .
16:35 tqrst semiosis: that page also advocates rm -rf $file
16:35 semiosis T0aD: there is a ,,(gfid resolver) that may help
16:35 glusterbot T0aD: https://gist.github.com/4392640
16:35 semiosis tqrst: yes thats how you resolve split brain, by deleting one of of the copies
16:35 tqrst semiosis: I'm not a huge fan of removing /
16:36 semiosis tqrst: best to prevent split brain before it happens
16:36 tqrst semiosis: how do you suggest I do that?
16:37 T0aD well i dont like this new quota patch
16:37 semiosis tqrst: quorum is probably the easiest way
16:37 tqrst (besides, / is already split brained, and rm -rf / is not an option, so I'm still stuck even if I figure out how to get 100% uptime on all my servers)
16:37 T0aD @split-brain
16:37 glusterbot T0aD: To heal split-brain in 3.3, see http://goo.gl/FPFUX .
16:40 T0aD semiosis, interesting, we should keep all those gists somewhere
16:40 semiosis tqrst: can you please pastie.org the logs from around the split-brain line?
16:42 tqrst semiosis: that's pretty much all there is, really: http://pastie.org/private/muysjma0nmpxkonwdwop2q
16:42 glusterbot <http://goo.gl/PvGzB> (at pastie.org)
16:42 semiosis T0aD: added a link here: http://www.gluster.org/community/docume​ntation/index.php/HowTo#Troubleshooting
16:42 glusterbot <http://goo.gl/8GNM9> (at www.gluster.org)
16:42 semiosis maybe not the best spot, but it's a start
16:43 semiosis tqrst: ah i see
16:43 lalatenduM joined #gluster
16:43 T0aD semiosis, i starred it to make a glusterfs network on github
16:43 semiosis tqrst: so here's what you'll want to do.....
16:43 semiosis look on all the bricks, at the brick root directory, and verify that the owners, perms, times, and entries are the same for that dir on all bricks
16:44 semiosis once you have confirmed then you can go in and manually reset the xattrs
16:44 T0aD semiosis, maybe you could put my link to tool to remove glusterfs xattrs  in the wiki
16:44 semiosis see ,,(extended attributes)
16:44 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
16:44 T0aD @remove-attributes
16:44 glusterbot T0aD: https://gist.github.com/T0aD/6004343
16:44 semiosis T0aD: it's a public wiki, feel free
16:44 T0aD i probably have to create an account and stuff
16:45 semiosis tqrst: you'd want to set the trusted.afr attribs on ALL BUT ONE of the bricks to 0
16:45 semiosis that will allow the heal to proceed
16:45 tqrst semiosis: does it matter which one?
16:45 semiosis if you've already manually verified that they're all the same, no
16:46 semiosis otherwise yes that one's properties will be synced to theothers
16:46 semiosis read that article about xattrs
16:47 mafrac Trying to upgrade to gluster 3.3.2 I get a signature error: http://pastie.org/8149617 . How could I solve it?
16:47 glusterbot Title: #8149617 - Pastie (at pastie.org)
16:53 mafrac And now with clusters in 3.4 I can't mount the filesystem: http://pastie.org/8149638
16:53 glusterbot Title: #8149638 - Pastie (at pastie.org)
16:53 tqrst semiosis: thanks, I'll give that a try after lunch
16:54 semiosis good luck
16:54 tqrst hopefully it's easy to automate, because there are quite a few :\
16:54 T0aD yeah its easy to play with xattrs
16:55 T0aD you can get my python script and play with it, it can list and remove attributes so far, without extra modules
16:59 mafrac Could somebody help me, please? All clients can't conect to glusters servers after upgrade to 3.4
16:59 T0aD mafrac, im afraid im not competent to do so
17:02 JonnyNomad I have a user that copied a 35GB file to his owncloud account via the desktop client. His quota is 20GB. He delete the file, but now there's a copy in his files_trashbin directory. Can I just delete that on the server or will that break stuff?
17:03 semiosis JonnyNomad: this is running on glusterfs?
17:03 mafrac T0aD, Thanks
17:03 JonnyNomad crap, sorry, wrong window
17:03 semiosis lol
17:04 semiosis mafrac: Started running /usr/sbin/glusterfs version 3.3.1 <-- your log.
17:04 T0aD mafrac, bad boy!
17:04 mafrac T0aD, why?
17:05 puebele1 joined #gluster
17:05 T0aD because i can mafrac, because i can.
17:06 mafrac Here is the log with the error mounting glusterfs: http://pastie.org/8149638
17:06 glusterbot Title: #8149638 - Pastie (at pastie.org)
17:06 mafrac ok, the client has 3.3.1 and the server 3.4
17:07 mafrac it is supposed compatible, true?
17:07 mafrac I'm trying to upgrade the clients with centos 5 to 3.3.2 but I get a signature error and it fails
17:08 mafrac http://pastie.org/8149617
17:08 glusterbot Title: #8149617 - Pastie (at pastie.org)
17:09 mafrac is it possible to upgrade centos 5 clients to 3.4 directly?
17:16 andreask joined #gluster
17:16 T0aD haha funny the rot13.c xlator
17:19 T0aD http://hekafs.org nice source of info
17:19 glusterbot Title: HekaFS (at hekafs.org)
17:23 bennyturns joined #gluster
17:23 mafrac Do you know where could I find if glusterfs 3.4 is compatible with centos 5.9?
17:30 bennyturns mafrac, I have use 5.9 clients alot but I haven't tried 5.9 as a server.  I don't know of any issues with running it as a server though.
17:31 mafrac Servers has centos 6.3 and glusterfs 3.4 now
17:31 jdarcy joined #gluster
17:31 mafrac clients are centos 5.9 only but with 3.3.1 they can't get volume information
17:32 mafrac bennyturns, thanks for your reply
17:32 bennyturns mafrac, ya, I got that before wehn accidentally trying to attach 3.3 clietns to 3.4 servers
17:32 bennyturns mafrac, you shloud be fine to upgrade em
17:33 T0aD alright, by tonight ill master the quotas
17:33 T0aD they ll be mine !
17:34 mafrac bennyturns, I made confident because they assumed that the 3.3.1 version was compatible with 3.4
17:34 mafrac but seems not
17:34 mafrac so, how could I upgrade to glusterfs 3.4 on centos 5.9 clients?
17:35 mafrac or should an upgrade to 3.3.2 work too? In this case, how could I solve the signature issue with yum update?
17:37 bennyturns mafrac, I think you woll need to upgrade the clientsx to 3.4 if you want to mount 3.4 servers.  Can you just grab the RPMs and install em with rpm -i?
17:37 bennyturns or u or whatever switches
17:38 mafrac I'll try
17:38 bennyturns mafrac, http://vbellur.wordpress.com/2013/​07/15/upgrading-to-glusterfs-3-4/
17:38 glusterbot <http://goo.gl/SXX7P> (at vbellur.wordpress.com)
17:39 mafrac bennyturns, I saw the post yet
17:39 _pol joined #gluster
17:39 mafrac but it don't help me
17:41 mafrac I get the same error with rpm -i : http://pastie.org/8149759
17:41 glusterbot Title: #8149759 - Pastie (at pastie.org)
17:42 rkeene joined #gluster
17:42 bennyturns mafrac, hrm, lemme try it on one of my RHEL 5 systems
17:42 rkeene I'm having a problem where Gluster simultaneously thinks my volume does not exist, and does exist.
17:42 rkeene http://www.rkeene.org/viewer​/tmp/gluster-r-dumb.txt.htm
17:42 glusterbot <http://goo.gl/yiIIj> (at www.rkeene.org)
17:44 bennyturns mafrac, I don't see any CENTOS 5 RPMs on http://download.gluster.org/pub/gl​uster/glusterfs/3.4/3.4.0/CentOS/
17:44 glusterbot <http://goo.gl/1XK04> (at download.gluster.org)
17:45 mafrac I know, does it mean that I can't update to 3.4 on centos 5?
17:45 bennyturns mafrac, I don't think thats the case, I think the RPMs just aren't built on centos 5
17:46 bennyturns mafrac, if you compile it from source it should work
17:47 mafrac well, I'll try
17:47 mafrac do yo know if I've to remove 3.3.1 first?
17:47 bennyturns mafrac, I usually remove the rpm before I install from source
17:48 mafrac on the other hand, what about upgrading to 3.3.2?
17:48 mafrac how could I solve the signature issue?
17:50 bennyturns mafrac, when I create yum repros for rhel 5 I need to use sha encrytion instead of the RHEL/centos 6 defaults.  maybe something to do with that?
17:51 yinyin joined #gluster
17:52 bennyturns mafrac, where did you get the rRPMS from?
17:53 mafrac http://download.gluster.org/pub/gluster/g​lusterfs/3.3/3.3.2/CentOS/epel-5/x86_64/
17:53 glusterbot <http://goo.gl/GBszc> (at download.gluster.org)
17:53 bennyturns kk lemme try those on my el5 system
17:53 daMaestro joined #gluster
17:54 T0aD hm hm i saw a lot of dict , wondering if gluster reused some python source code :)
17:54 T0aD like dict_new()
17:55 JoeJulian mafrac, bennyturns: I think there are prerequisites that are not met in EL5. There's also some significant fuse deficiencies.
17:56 bennyturns JoeJulian, oh for 3.4 you mena?
17:56 JoeJulian Right
17:56 bennyturns JoeJulian, oh!  I didn't know that
17:56 mafrac I too
17:58 mafrac Then, are 3.3.2 clients compatible with 3.4 servers?
17:58 tqrst they're supposed to be
17:58 mafrac and 3.3.1 -> 3.4?
17:58 JoeJulian I haven't tried that direction, but like tqrst said, they're supposed to be.
17:58 skyw joined #gluster
17:59 JoeJulian But if it will compile, there's no reason not to use the 3.4 client on el5. It will perform just as miserably as 3.3 without the current version of fuse.
17:59 mafrac For me it doesn't matter stay with 3.3.1 clients -> 3.4 servers but I can't connect now
18:00 tqrst JoeJulian: do you have a version number for what you consider to be current, or is it just whatever is available post el5?
18:00 dpaul joined #gluster
18:00 tqrst just curious if what we have is recent enough
18:05 JoeJulian tqrst: I don't. Look for kernel patches submitted by avati. I would, but I've got a splitting headache and $dayjob to do...
18:05 * tqrst nods
18:06 mafrac bennyturns, JoeJulian,  then, how could I solve the "failed to fetch volume file" issue at mounting?
18:06 raghug joined #gluster
18:07 T0aD hi JoeJulian :)
18:10 JoeJulian el5 rpms are supposed to be signed with md5, so that's probably why that fails... You could probably install if you set nogpgcheck
18:11 JoeJulian mafrac: Paste the glusterd.vol.log for that mount attempt
18:11 mafrac JoeJulian, at client or server?
18:11 JoeJulian etc-glusterfs-glusterd.vol.log is on the server.
18:11 mafrac thanks
18:11 JoeJulian That's the log file for glusterd.
18:11 JoeJulian @processes
18:12 glusterbot JoeJulian: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/F6jqx for more information.
18:13 mafrac [2013-07-17 18:14:18.499260] I [glusterd-handshake.c:194:_client_supports_volume] 0-glusterd: Client 91.121.225.182:1023 (1 -> 1) doesn't support required op-version (2). Rejecting volfile request.
18:13 raghug_ joined #gluster
18:13 mafrac sorry for sending the ip
18:14 T0aD the last forks:
18:14 T0aD T facebook / glusterfs
18:14 T0aD L T0aD / glusterfs
18:14 T0aD :D
18:14 mafrac JoeJulian, [2013-07-17 18:14:18.499260] I [glusterd-handshake.c:194:_client_supports_volume] 0-glusterd: Client x.x.x.x:1023 (1 -> 1) doesn't support required op-version (2). Rejecting volfile request.
18:15 JoeJulian Well, poo.
18:15 JoeJulian cd
18:15 JoeJulian @meh
18:15 glusterbot JoeJulian: I'm not happy about it either
18:18 T0aD haha
18:20 mafrac JoeJulian, any idea how to solve it?
18:20 edong23 joined #gluster
18:22 aliguori joined #gluster
18:22 JoeJulian several... trying one now...
18:29 Recruiter joined #gluster
18:29 _pol joined #gluster
18:32 mafrac JoeJulian, thanks
18:33 chirino joined #gluster
18:36 krink_ joined #gluster
18:37 Hoggins joined #gluster
18:38 Hoggins hello everyone, I have a very simple problem I guess... testing Gluster, and during a file copy, the client server crashed... it was voluntary. Anyway, now I have a split-brain condition on a file... but I can't figure out what to do to solve it. It's a replicated volume amongst two nodes.
18:39 Hoggins (running Gluster 3.3)
18:42 JoeJulian ~split-brain | Hoggins
18:42 glusterbot Hoggins: To heal split-brain in 3.3, see http://goo.gl/FPFUX .
18:42 JoeJulian @change split-brain s/3.3/3.3+/
18:42 glusterbot JoeJulian: Error: The command "change" is available in the Factoids, Herald, and Topic plugins. Please specify the plugin whose command you wish to call by using its name as a command before "change".
18:42 JoeJulian @factoids change split-brain s/3.3/3.3+/
18:42 glusterbot JoeJulian: Error: 's/3.3/3.3+/' is not a valid key id.
18:42 JoeJulian @factoids change split-brain 1 s/3.3/3.3+/
18:42 glusterbot JoeJulian: The operation succeeded.
18:43 Hoggins OK, my bad, I had followed JoeJulian's guide, and I did not figure out that it had cured my I/O error problem
18:44 Hoggins the split-brain condition is still displayed by the "heal split-brain" command, though
18:44 JoeJulian Yeah, it's more of a log than a status indication. Note the timestamps.
18:44 JoeJulian We filed a bug on that ages ago.
18:44 theron joined #gluster
18:44 Hoggins oh okay, I had not understood this important detail about the "log" aspect :)
18:45 Hoggins thanks guys, everything is okay now
18:45 JoeJulian excellent
18:46 JoeJulian mafrac: http://koji.fedoraproject.org​/koji/taskinfo?taskID=5622425
18:46 glusterbot <http://goo.gl/wpyjT> (at koji.fedoraproject.org)
18:47 Hoggins see ya, and thanks again !
18:47 krink joined #gluster
18:47 raghug joined #gluster
18:48 mafrac JoeJulian, Thanks!!
18:50 mafrac JoeJulian, Then can I install it? how?
18:51 JoeJulian Copy the link for the rpms you want to install, and rpm -Uvh <link> [<link> ...]
18:54 mafrac JoeJulian, It works!! Thank you a lot!1
18:55 JoeJulian You're welcome
18:55 mafrac Have a good day JoeJulian
18:59 T0aD hip hip hip ! JoeJulian !
19:09 sjoeboo so, just noticed, no el5 packages for 3.4 yet?
19:15 mafrac until now, thanks to JoeJulian
19:21 glusterbot New news from newglusterbugs: [Bug 985565] Wrong source macro referenced for el5 build <http://goo.gl/vrcXn>
19:30 mafrac JoeJulian, I think that I've the same problem with debian wheezy
19:31 roidelapluie some reading: http://roidelapluie.be/glusterfs-split-brain.html
19:31 glusterbot <http://goo.gl/zpveh> (at roidelapluie.be)
19:31 mafrac I've some debian wheezy clients with 3.3.1
19:32 JoeJulian kkeithley: Apparently your builds are not meeting with the infamous ,,(Joe's performance metric) .
19:32 glusterbot kkeithley: nobody complains.
19:32 jdarcy joined #gluster
19:32 plarsen joined #gluster
19:32 Myk joined #gluster
19:33 JoeJulian kkeithley: If you do want to build for epel5, you'll need http://review.gluster.org/5339 (bug 985565)
19:33 glusterbot Bug http://goo.gl/vrcXn unspecified, unspecified, ---, amarts, NEW , Wrong source macro referenced for el5 build
19:40 yinyin joined #gluster
19:41 kkeithley I wasn't planning on building 3.4 or el5
19:42 kkeithley s/or el5/for el5/
19:42 glusterbot What kkeithley meant to say was: I wasn't planning on building 3.4 for el5
19:43 JoeJulian sjoeboo: Looks like you'll have to install from my scratch build then: http://koji.fedoraproject.org​/koji/taskinfo?taskID=5622425
19:43 glusterbot <http://goo.gl/wpyjT> (at koji.fedoraproject.org)
19:44 sjoeboo JoeJulian: "scratch" build in terms of what?
19:44 sjoeboo 3.4 for el is critical to us...can't have mixed clients out there,a and we got a couple 3.4 deployments in testing/planning.
19:44 JoeJulian In terms of it's not going to end up in the epel repo
19:45 sjoeboo el5*
19:45 sjoeboo ah, thats okay, i actually jsut want a working SRPM i can build against our stakc to be double extra sure.
19:45 JoeJulian The srpm is there too.
19:45 sjoeboo coolio
19:46 sjoeboo hm, hasn't built yet?
19:46 sjoeboo no output...
19:49 Myk Has anyone run into an issue where you cannot see files via nfs mount (they do exist on the file system, and can be manipulated by name) but do not show up via nfs?
19:50 andreask joined #gluster
19:50 JoeJulian sjoeboo: http://kojipkgs.fedoraproject.org//work/task​s/2427/5622427/glusterfs-3.4.0-2.el5.src.rpm
19:50 glusterbot <http://goo.gl/hNxki> (at kojipkgs.fedoraproject.org)
19:50 sjoeboo hm, went trying to rebuild that SRPM, i getfailure and lots of no such file
19:51 JoeJulian Wierd. It (obviously) built under the standard build environment. I wonder what's unique about yours.
19:52 JoeJulian Myk: Never heard of that. Check the nfs.log and/or the brick logs for errors.
19:52 JoeJulian Myk: Also try mounting via fuse and see if that's any different.
19:53 sjoeboo JoeJulian: Not much, beyond us locking yum updates @ specific point in time intervals, there isn't much special
19:53 sjoeboo its litterally all:
19:53 sjoeboo Bad file: /usr/src/redhat/SOURCES/object-expirer.conf: No such file or directory
19:53 sjoeboo Bad file: /usr/src/redhat/SOURCES/swift.conf: No such file or directory
19:53 sjoeboo etc
19:53 sjoeboo all swift related (which i don't need)
19:54 kkeithley 3.3.2/epel-5 repo is fixed
19:55 kkeithley any other problems I should know about?
19:55 kkeithley did glusterbot show you my comment about adding the Before=network-online.target in glusterd.service works for me. The trick seems to be not using _netdev in your /etc/fstab. If I change defaults to _netdev then it doesn't work. (defaults,_netdev seems to work too)
19:57 JoeJulian Interesting, though I think the _netdev thing is a red herring and it's all timing related.
19:57 _pol joined #gluster
19:57 joelwallis joined #gluster
19:58 kkeithley could be. I only tried w/ a vm.
19:58 joelwallis left #gluster
19:58 dbruhn Whats the command to list all of the bricks a file is stored on?
19:58 mooperd joined #gluster
20:01 semiosis ~pathinfo | dbruhn
20:01 glusterbot dbruhn: find out which brick holds a file with this command on the client mount point: getfattr -d -e text -n trusted.glusterfs.pathinfo /client/mount/path/to.file
20:01 T0aD nice
20:01 T0aD ~quota | T0aD
20:01 glusterbot T0aD: I do not know about 'quota', but I do know about these similar topics: 'quorum'
20:01 T0aD pff.
20:02 piotrektt hi. can you tell me how distribution of files will look in this setup? according to gluster administration guide i should have the same file on 101 and 102. is that correct? http://pastebin.com/ELAuerqb
20:02 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
20:02 piotrektt @paste
20:02 glusterbot piotrektt: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
20:07 dbruhn thanks semiosis
20:07 semiosis yw
20:08 piotrektt hey. did my question appeared or was it blocked by bot?
20:13 kkeithley tqrst, coredumb: epel-6 repos fixed for CentOS and Scientific. So much for them being RHEL clones.
20:15 JoeJulian piotrektt: The bot doesn't block anything. It's simply there to provide additional help.
20:16 JoeJulian ~brick-order | piotrektt
20:16 glusterbot piotrektt: I do not know about 'brick-order', but I do know about these similar topics: 'brick order'
20:16 JoeJulian ~brick order | piotrektt
20:16 glusterbot piotrektt: Replicas are defined in the order bricks are listed in the volume create command. So gluster volume create myvol replica 2 server1:/data/brick1 server2:/data/brick1 server3:/data/brick1 server4:/data/brick1 will replicate between server1 and server2 and replicate between server3 and server4.
20:17 piotrektt so the sysadmin guide is making confusion
20:18 piotrektt i have it wrong then :(
20:18 piotrektt craps
20:18 piotrektt http://screencloud.net/v/dh0i
20:18 glusterbot Title: Screenshot at 22:18:32 | ScreenCloud (at screencloud.net)
20:21 piotrektt ok, now I get it, but the picture is misinforming :(
20:21 T0aD piotrektt, how come ?
20:22 piotrektt T0aD it should be shown on the picture that the replica set is created beetween exp1 and exp2 and it is beetween exp1 and exp3
20:22 piotrektt it does not state what the command was
20:22 piotrektt at first glance it can misinform
20:23 piotrektt because the command later on in the guide is to create with order of exp1 exp2 exp3 exp4
20:24 piotrektt and the worst part is I need to redo it ;(
20:24 T0aD ohlala!
20:26 tqrst kkeithley: great, thanks
20:27 tqrst (and I agree that $releasever shouldn't be touched when you claim to be a clone...)
20:32 piotrektt ok. thanks for help. now I have it the right way :)
20:35 _pol joined #gluster
20:39 T0aD ah cool
20:39 T0aD there is  a nice .gitignore in da git :)
20:39 chirino joined #gluster
20:47 coredumb kkeithley: Ok thx
20:47 coredumb wouldn't have been easier to just have 6 instead of $releasever in .repo ?
20:50 jdarcy joined #gluster
20:52 Myk joejulian: thanks for the headsup, looks like a version mismatch (3.0.5 client versus 3.2.5 on server)
20:56 jdarcy_ joined #gluster
20:57 FinnTux joined #gluster
21:04 FinnTux will debian repository have 3.4 anytime soon?
21:04 semiosis yes, working on that
21:05 FinnTux great, thanks
21:11 FinnTux Is it possible to define bricks using different IPs the nodes have? To maximize bandwidth usage (2x1Gbit/s 802.3ad)?
21:12 FinnTux I have 2 nodes with 3 disks each I like to use as bricks
21:14 FinnTux "peer probe 2nd.ip.of.host" says host is already in peer list but yet it can't be used.
21:15 _pol joined #gluster
21:21 piotrektt it would be more readable if you paste the output command line gives you :)
21:28 yinyin joined #gluster
21:38 FinnTux right, I'll get back into that when I test gluster again
21:39 FinnTux but is there a way to distribute network traffic across different links using two nodes?
21:42 chirino joined #gluster
21:43 JoeJulian I guess that depends on what a node is and how your network is built.
21:43 jebba joined #gluster
21:45 FinnTux node=computer. Two servers with 2x1G nics connected to a switch. Two (or more if needed) vlans for gluster traffic
21:45 FinnTux bonding mode 802.3ad
21:46 duerF joined #gluster
21:46 JoeJulian To be clear, clients are computers too. :P
21:46 FinnTux yup and in this case servers are also clients
21:49 JoeJulian Then I guess I'm not interpreting your question very well (possible as this damned headache just won't quit today). If you have two servers/clients and you have bonded ethernet through a switch between them, what is there to be distributed?
21:50 FinnTux 3 disk (=3 bricks) on each server
21:50 FinnTux disks*
22:01 foster joined #gluster
22:02 FinnTux right, now I have 3 directories ready to be used as bricks
22:03 FinnTux lets say I create 1st volume:
22:03 FinnTux gluster volume create gvol_sdc 172.16.0.172:/export/sdc 172.16.0.173:/export/sdc
22:03 FinnTux Creation of volume gvol_sdc has been successful. Please start the volume to access data.
22:04 FinnTux fine. everything is ok. now I'd like to use another vlan (172.16.1.172 and .173) for second volume
22:04 FinnTux gluster volume create gvol_sdd 172.16.1.172:/export/sdd 172.16.1.173:/export/sdd
22:04 FinnTux Host 172.16.1.173 not a friend
22:05 FinnTux probing won't work
22:05 FinnTux gluster peer probe 172.16.1.173
22:05 FinnTux Probe on host 172.16.1.173 port 0 already in peer list
22:09 FinnTux forgot "replica 2" from create command...
22:10 T0aD funny
22:11 T0aD why dont quota_parse_limits() isnt called right away when i start glusterfsd ?
22:15 T0aD but the quota works hm hm...
22:22 JoeJulian FinnTux: Will that actually balance anything? It's all going to use the same hardware. The only way to get the behavior you're looking for, that I can think of, is with iptables. The services listen on 0.0.0.0 and on a single box, even if you use hostnames there's no way to distinguish that hostname:port should be different for one volume vs another.
22:24 JoeJulian There has been discussion about adding more advanced network features, perhaps in the next version.
22:24 FinnTux I tested drbd and traffic was balanced between 1Gbit/s links (3 disks - 3 drbds - 3 vlans).
22:25 JoeJulian Then use drbd.
22:25 FinnTux it has its own problems...
22:26 JoeJulian I've met them.... <eyeroll>
22:27 FinnTux it's pretty confusing that vol create says "Host xx not a friend" but trying to probe it says "Probe on host xx port 0 already in peer list"
22:27 FinnTux so is it a friend or not...
22:27 bobby_ joined #gluster
22:29 JoeJulian Right. The hostname or ip address is converted to a uuid that identifies that server. When you try to create a volume with a server that cannot be identified, you get the "not a friend". When you probe the ip address, the target server returns its uuid. That uuid is a peer, so you get the second error.
22:29 bobby_ I'm having an issue I'm wondering if anyone can lead me in the right direction on.
22:29 JoeJulian That's one of two possibilities, bobby_
22:29 kedmison joined #gluster
22:29 bobby_ We had a power failure during a brick-replace and now all the brick-replace commands fail
22:30 JoeJulian even abort?
22:30 bobby_ Joe, yes unfortunately so
22:30 JoeJulian abort force?
22:31 FinnTux ok. thanks JoeJulian
22:31 JoeJulian Use the force.... <loud mask breathing>
22:31 bobby_ Didn't try abort force, will do so now and report back
22:31 bobby_ Thank you JOE
22:31 JoeJulian Good luck
22:34 bobby_ Joe, It tells me, Volume: Command not found
22:34 mafrac JoeJulian, I've the same problem that bobby_ but with abort force I get the "Usage: volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start|pause|abort|status|commit [force]}" message
22:34 nightwalk joined #gluster
22:34 bobby_ Sorry, blonde moment
22:35 JoeJulian Ah, ok... so it's only commit that can be forced, eh?
22:35 JoeJulian darn.
22:36 mafrac what happens if i do commit force without move data?
22:36 mafrac is data accesible by the client?
22:36 JoeJulian I guess that makes some sense. If the data is only half migrated, it would be possible for some of the files to be out of sync, creating a split-brain situation if you were to try to start it again. I think "commit force" may be your only option.
22:37 JoeJulian If this is a replicated volume, you should be fine. If not, you'll have to manage migrating any missing (or damaged) files by hand.
22:38 bobby_ It is a mirrored volume
22:38 bobby_ So if I commit force, then it will resync my partically copied drive from my fully correct drive?
22:39 bobby_ Data loss won't be an issue I need to be concerned about?
22:41 JoeJulian no worries. I would do a "gluster volume heal $vol full"
22:42 edong23_ joined #gluster
22:43 bobby_ Should I make a backup of the good drive just for poops and giggles?
22:45 mafrac Indeed some files have disappeared after commit force
22:46 JoeJulian You can. The "commit force" method of brick replacement in a replicated scenario was considered as the default. The only reason it's not is because some of us are belt and suspenders folks. We would rather have the data in 3 places temporarily than just in one.
22:47 JoeJulian mafrac: What version is that? I've done this successfully 60 times.
22:47 mafrac but the data lost is still on server removed
22:47 bobby_ I definitely would rather have it in 3 places.  A loss of data would suck for us
22:47 mafrac recently upgraded to 3.4
22:47 [o__o] joined #gluster
22:48 piotrektt joined #gluster
22:48 mafrac with replica 0
22:48 jag3773 joined #gluster
22:48 mafrac what is the best way to sync the lata lost?
22:48 kevein joined #gluster
22:49 JoeJulian Depends on a number of factors. I'd use rsync to a client mount if I knew that the source data was the good data.
22:49 JoeJulian make sure to use --inplace so you don't get a bunch of sticky pointers.
22:50 Recruiter joined #gluster
22:53 badone joined #gluster
22:54 mafrac Is ther any way to know if there is any job (rebalance, remove-brick...) over any volume?
22:55 JoeJulian Good question. If there's not there should be. "volume status" or "volume info" are the two most likely candidates.
22:59 mafrac Thanks
23:01 nightwalk joined #gluster
23:08 bala joined #gluster
23:13 matiz joined #gluster
23:14 tom_cruise joined #gluster
23:14 tom_cruise Hello!!!
23:14 glusterbot tom_cruise: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
23:14 coredumb joined #gluster
23:14 T0aD tom_cruise  is using gluster !  I knew it !
23:15 tom_cruise Can you please help me replace a failed server in a distributed-replicated setup?
23:15 T0aD glusterbot, keep it short, will ya
23:16 tom_cruise If you're in the mailing list, I posted it under the subject "I Need Help Replacing a Failed Server"
23:16 masterzen joined #gluster
23:17 tom_cruise I followed this http://gluster.org/community/documen​tation/index.php/Gluster_3.2:_Brick_​Restoration_-_Replace_Crashed_Server
23:17 glusterbot <http://goo.gl/60uJV> (at gluster.org)
23:17 tom_cruise But that doesn't work.
23:17 tom_cruise I'm on 3.4 on CentOS 6 x86-64
23:18 tom_cruise The only documentation I am finding concerning this is the link I posted above. The rest is replacing only a brick, not the entire server.
23:19 tom_cruise What information can I volunteer to help you help me?
23:19 tom_cruise Anyone?
23:30 mafrac What happened to failed server?
23:31 T0aD he played in an action movie
23:31 mafrac :D
23:32 mafrac tom_cruise, I think that you can detach the brick, add a new one and do a rebalance. Has yor tried it?
23:33 mafrac no problem if volume is replicated
23:34 tom_cruise correct volume is replicated therefore the brick cannot be detached.
23:35 tom_cruise I purposely destroyed the server.
23:35 tom_cruise This is an evaluation period for my company.
23:35 tom_cruise I have evaluated all other things we would need from something like this, but it is this specific case that I cannot resolve.
23:36 tom_cruise Did you see my email from the mailing list?
23:38 mafrac not yet
23:38 mafrac have you tried detach force?
23:39 tom_cruise I explained everything in that email. All the actions I have taken.
23:39 tom_cruise I'm using vagrant so I am able to recreate my test environment quickly.
23:39 tom_cruise I can destroy and rebuild everything again and it will predictably fail only on this scenario.
23:39 tom_cruise So I just finished rebuilding.
23:40 tom_cruise I'm going to recreate the problem line by line.
23:41 tom_cruise for i in 2 3 4;do gluster peer probe gluster${i}; done
23:41 tom_cruise Now I have four servers in the trusted pool.
23:41 JoeJulian tom_cruise... have you contacted your IMF team? This should be trivial for them. ;)
23:41 tom_cruise [root@gluster1 ~]# gluster peer status
23:41 tom_cruise Number of Peers: 3
23:41 tom_cruise Hostname: gluster2
23:41 tom_cruise Port: 24007
23:41 tom_cruise Uuid: 5a9ffca7-7399-49a4-968c-0de418770129
23:41 tom_cruise State: Peer in Cluster (Connected)
23:41 tom_cruise Hostname: gluster3
23:41 tom_cruise Port: 24007
23:41 tom_cruise Uuid: 3bdb0c09-eb85-46d0-bd02-f5881f7a3b70
23:41 tom_cruise State: Peer in Cluster (Connected)
23:42 tom_cruise Hostname: gluster4
23:42 tom_cruise Port: 24007
23:42 JoeJulian gah!
23:42 tom_cruise Uuid: a2422293-42c7-4bd0-a6cc-2a92674448a9
23:42 tom_cruise State: Peer in Cluster (Connected)
23:42 JoeJulian tom_cruise: Don't paste in channel please. ,,(paste))
23:42 glusterbot tom_cruise: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
23:42 T0aD whats IMF ?
23:42 JoeJulian (Impossible Mission Force)
23:43 T0aD ohlala
23:43 al joined #gluster
23:45 tom_cruise gluster volume create vol01 replica 2 gluster{1,2,3,4}:/data
23:45 tom_cruise That creates me vol01
23:46 tom_cruise [root@gluster1 ~]# gluster volume info
23:46 tom_cruise
23:46 tom_cruise Volume Name: vol01
23:46 tom_cruise Type: Distributed-Replicate
23:46 tom_cruise Volume ID: 48befd61-4f96-4aed-8417-0b3ca15dfa7d
23:46 tom_cruise Status: Started
23:46 tom_cruise Number of Bricks: 2 x 2 = 4
23:46 tom_cruise Transport-type: tcp
23:46 tom_cruise Bricks:
23:46 tom_cruise Brick1: gluster1:/data
23:46 tom_cruise Brick2: gluster2:/data
23:46 tom_cruise Brick3: gluster3:/data
23:46 tom_cruise Brick4: gluster4:/data
23:47 tom_cruise So that volume is already started...
23:47 T0aD tom_cruise, can you please run 'gluster volume info vol01' and show us the result ?
23:47 tom_cruise Remember, the problem hasn't surfaced yet. I just rebuilt my test environment...
23:47 JoeJulian tom_cruise: Repeat: No pasting more than 3 lines in channel.
23:48 T0aD tom_cruise, i still think thats important
23:48 tom_cruise [root@gluster1 ~]# gluster volume info vol01
23:48 tom_cruise
23:48 tom_cruise Volume Name: vol01
23:48 tom_cruise Type: Distributed-Replicate
23:48 tom_cruise Volume ID: 48befd61-4f96-4aed-8417-0b3ca15dfa7d
23:48 tom_cruise Status: Started
23:48 tom_cruise Number of Bricks: 2 x 2 = 4
23:48 tom_cruise Transport-type: tcp
23:48 tom_cruise Bricks:
23:48 tom_cruise Brick1: gluster1:/data
23:48 tom_cruise Brick2: gluster2:/data
23:48 tom_cruise Brick3: gluster3:/data
23:48 tom_cruise Brick4: gluster4:/data
23:48 JoeJulian @kick tom_cruise
23:48 tom_cruise was kicked by glusterbot: JoeJulian
23:48 T0aD haha.
23:48 tom_cruise joined #gluster
23:48 JoeJulian He's not banned. He can come back.
23:49 tom_cruise JoeJulian, I'm looking for help, not aggravation.
23:49 JoeJulian Then pay attention.
23:49 tom_cruise I'm not spamming.
23:49 tom_cruise It was an honest request for information and it was given.
23:49 T0aD tom_cruise, try using http://bpaste.net
23:49 JoeJulian I help, I don't like aggravation either.
23:49 glusterbot Title: New Paste at spacepaste (at bpaste.net)
23:49 T0aD I rarely help and often look for aggravation
23:49 tom_cruise Joe, you just aggravated me.
23:49 T0aD damn I said it
23:50 JoeJulian And spamming 30 lines into a chat window is aggravating.
23:50 tom_cruise Joe, it's spam if it's unsolicited.
23:50 JoeJulian Follow directions. That's all I ask.
23:50 JoeJulian @channelstats
23:50 glusterbot JoeJulian: On #gluster there have been 157688 messages, containing 6692058 characters, 1121920 words, 4575 smileys, and 598 frowns; 1011 of those messages were ACTIONs. There have been 59255 joins, 1851 parts, 57445 quits, 20 kicks, 157 mode changes, and 6 topic changes. There are currently 171 users and the channel has peaked at 217 users.
23:51 JoeJulian 20 kicks in 3 years.
23:51 JoeJulian 1 ban
23:51 tom_cruise That rule is capricious and based on zero logic.
23:51 tom_cruise But I'm not here to argue that.
23:52 tom_cruise If I can't get assistance here, where can I go?
23:52 JoeJulian Here's the quick version, T0aD http://blog.gmane.org/gmane.c​omp.file-systems.gluster.user
23:52 glusterbot <http://goo.gl/m81y6> (at blog.gmane.org)
23:52 JoeJulian Nobody said you can't. We just ask you to use a paste site for pasting.
23:53 JoeJulian @paste
23:53 T0aD JoeJulian, im busy trying to find where the quota limits are fetched from :)
23:53 glusterbot JoeJulian: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
23:54 JoeJulian In your email, which one was the "failed" server?
23:54 JoeJulian Oh, nm... 2
23:55 JoeJulian Which one is "192.168.56.204"?
23:55 tom_cruise 204 is node 4.
23:55 tom_cruise node1 == 1
23:55 tom_cruise node2 == 2
23:55 tom_cruise etc...
23:56 JoeJulian Sure... I presume you originally probed all your peers from 4. You can fix that to reference the hostname by probing back to 4 by hostname from any other server
23:56 tom_cruise Yes, but I killed 2.
23:57 tom_cruise 4 is fine.
23:57 JoeJulian Sure, just something I noticed is all...
23:57 tom_cruise So I changed the glusterd.info file to have the UUID that all the other servers expect it.
23:57 tom_cruise I just rebuilt the environment and am able to recreate the issue by destroying 2.
23:57 tom_cruise And then recreating it.
23:58 JoeJulian I've got to go catch a train. Are you going to be around in about an hour and a half or so? I should be home and on a computer by then.
23:59 tom_cruise I think so.
23:59 tom_cruise I have tried doing the peer probe from any of the random four servers...
23:59 tom_cruise The problem is once I rebuild and change the UUID of 2...

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary