Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-03

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:21 StarBeast joined #gluster
01:01 mjrosenb does anyone happen to know how to get find to find all files with permissions 000?
01:05 * mjrosenb is assuming that find -perm 0 will work.
01:10 majeff joined #gluster
01:18 _pol joined #gluster
01:35 portante joined #gluster
02:39 wgao joined #gluster
04:13 hchiramm_ joined #gluster
04:16 isomorphic joined #gluster
04:20 majeff joined #gluster
04:29 sgowda joined #gluster
04:31 anands joined #gluster
04:33 hjmangalam1 joined #gluster
04:35 hchiramm_ joined #gluster
04:36 aravindavk joined #gluster
04:36 shylesh joined #gluster
04:40 vpshastry joined #gluster
04:45 mohankumar__ joined #gluster
04:50 brunoleon joined #gluster
04:53 kevein joined #gluster
05:10 psharma joined #gluster
05:18 rastar joined #gluster
05:21 hagarth joined #gluster
05:26 rotbeard joined #gluster
05:27 vpshastry joined #gluster
05:30 satheesh joined #gluster
05:30 satheesh1 joined #gluster
05:32 lalatenduM joined #gluster
05:34 guigui3 joined #gluster
05:40 guigui3 left #gluster
05:40 lalatenduM joined #gluster
05:45 guigui3 joined #gluster
05:50 majeff joined #gluster
06:02 sgowda joined #gluster
06:06 vimal joined #gluster
06:22 jtux joined #gluster
06:23 glusterbot New news from newglusterbugs: [Bug 952029] Allow an auxiliary mount which lets users access files using only gfids <http://goo.gl/x5z1R>
06:37 majeff joined #gluster
06:37 sgowda joined #gluster
06:42 dobber_ joined #gluster
06:44 jtux joined #gluster
06:52 krishna joined #gluster
06:56 StarBeast joined #gluster
07:00 ctria joined #gluster
07:02 brunoleon___ joined #gluster
07:17 hybrid512 joined #gluster
07:19 bala joined #gluster
07:24 andreask joined #gluster
07:25 rb2k joined #gluster
07:27 tjikkun_work joined #gluster
07:28 ricky-ticky joined #gluster
07:32 majeff joined #gluster
07:43 masterzen joined #gluster
07:55 rb2k I get a lot of warning messages like these
07:55 rb2k "remote operation failed: File exists (00000000-0000-0000-0000-000000000000 -> /home/slowlog/prod)"
07:55 rb2k is that normal?
07:55 rb2k especially all the 0s as the gifd?
07:56 raghu joined #gluster
08:18 majeff joined #gluster
08:25 majeff1 joined #gluster
08:28 hchiramm_ joined #gluster
08:31 Norky joined #gluster
08:34 saurabh joined #gluster
08:36 spider_fingers joined #gluster
08:56 kevein joined #gluster
09:02 kshlm joined #gluster
09:16 ollivera joined #gluster
09:19 deepakcs joined #gluster
09:34 bala joined #gluster
09:38 rastar joined #gluster
09:42 StarBeast joined #gluster
09:50 ThatGraemeGuy joined #gluster
10:14 vpshastry joined #gluster
10:14 saurabh joined #gluster
10:15 guigui1 joined #gluster
10:15 hagarth joined #gluster
10:21 yinyin joined #gluster
10:26 duerF joined #gluster
10:36 duerF joined #gluster
10:38 rastar joined #gluster
10:40 js_ will a replicated volume perform like shit in general on two aws micro instances?
10:44 ujjain joined #gluster
10:46 lh joined #gluster
10:46 lh joined #gluster
10:51 kke can i stop writing to a brick until layout is fixed?
10:51 kke or trying
10:51 kke i think there's something wrong on the new ones
10:51 kke getting no space left on device
10:57 kke [2013-06-03 13:30:16.933508] W [dht-selfheal.c:287:dht_selfheal_dir_mkdir_cbk] 0-m1_data-dht: selfhealing directory /tmp/BK7oXNPje failed: No space left on device
10:58 kke [2013-06-03 13:30:16.955174] I [client3_1-fops.c:289:client3_1_mkdir_cbk] 0-m1_data-client-3: remote operation failed: No space left on device
11:00 hagarth1 joined #gluster
11:08 edward1 joined #gluster
11:11 jiku joined #gluster
11:29 rwheeler joined #gluster
11:45 hagarth joined #gluster
11:56 yinyin_ joined #gluster
12:10 kke i can write data mostly just fine, but making dirs seems to fail constantly
12:17 bennyturns joined #gluster
12:17 Norky kke, my guess as that you have run out of inodes on one or more bricks
12:17 hchiramm__ joined #gluster
12:18 Norky do a "df -i" on the underlying brick FSes on the server(s)
12:22 kke let's see
12:22 kke yeah
12:22 kke /dev/vdd1            5242880 5242880       0  100% /export/data_2
12:22 kke can i add more? it's ext4
12:23 kke seems like no
12:23 Norky I'm pretty sure, no
12:23 kke hmmmmm
12:23 kke what the heck should i do now then
12:24 Norky I'm guessing you have a lot of small or zero-length files?
12:24 kke plenty of small files yes
12:24 kke or actually a lot of directories might be the problem
12:24 Norky yes, directories are "small files" in that sense :)
12:25 Norky number of inodes are set at mkfs time, either with a direct parameter or with one of the meta options like "small files" which will pick appropriate numbers
12:25 kke looks like if you increase the size, you can increase inode count
12:25 kke should've used reiser or something
12:25 kke or tweaked when doing mkfs
12:26 Norky err, I think thujat is an option, if it is on an LVM which you can epxand, but if you then put more small files onto it you'll have the same problem later....
12:26 guigui1 joined #gluster
12:27 kke yeah
12:27 nightwalk joined #gluster
12:27 Norky XFS is the current recommended brick XFS, ext4 has a recent change which can cause a problem for GLuster
12:27 kke so i guess the solution is to create new bricks with xfs and copy the stuff over
12:27 kke looks like all my bricks are nearing 100%
12:28 kke too bad i'm also running out of space in the volume group
12:29 Norky vdd1 is a 'raw' disk in a virtual machine, which is backed by LVM VGs on the host?
12:29 Norky you'll want the "-i size=512" option for XFS, but that's covered in the Gluster docs
12:30 Norky some people DO use ext4 successfully, and indeed ZFS
12:30 Norky not sure about reiser
12:31 Norky but the normal recommendation is XFS - your choice
12:31 kke mkfs.ext4 -I 512 /dev/vde1
12:31 kke tune2fs -i0 -c0 -o journal_data_writeback /dev/vdi1
12:31 kke those are the commands i've been using to create fs's
12:32 plarsen joined #gluster
12:33 Norky mkfs.ext4 -i 8192  (or even 4096 in the extreme case)      will give you more inodes
12:34 kke but 512 for xfs?
12:34 bennyturns joined #gluster
12:35 Norky ext4 -I 512    is, I think, equivalent to      xfs -i size=512
12:36 Norky xfs uses a different structure to ext2/3/4, so it's not directly comparable
12:36 Norky I don't fully understand it
12:36 Norky if you want to use ext4, see http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
12:36 glusterbot <http://goo.gl/PEBQU> (at joejulian.name)
12:37 Norky to repeat the 'party line': "N.B. We (Red Hat, Gluster) generally recommend using xfs for brick volumes; or if you’re feeling brave, btrfs. If you’re using ext4 be aware of the ext4 issue* and if you’re using ext3 make sure you mount it with -o user_xattr." :)
12:37 ProT-0-TypE joined #gluster
12:38 kke but just to be safe, should i use larger number with xfs? i have about 5-10 million files in almost as many directories (maybe 80%)
12:38 kke let's see what btrfs is, i'm brave :)
12:38 kke It is still in heavy development and marked as unstable
12:38 kke ok let's not
12:38 harish joined #gluster
12:39 Norky note that your current probl;em is nothing to do with gluster - it's simply a limitation of ext2/3/4 and many small files. You woudl have encountered it simply trying to store those files on a local ext FS withotu gluster being involved
12:39 kke yeah
12:39 Norky yeah, btrfs is not production ready
12:40 Norky the easy answer is just to use XFS :)
12:40 Norky but what distro are you running?
12:41 kke ubuntu 11.10 (should upgrade that too some day)
12:41 majeff joined #gluster
12:42 kke virtual machines managed by ganeti, so there's a vg where all the lvm's for vm's are living
12:42 harish joined #gluster
12:43 Norky AFAIK Ubuntu supports XFS
12:44 kke yeah it does, i have some xfs'es
12:45 majeff1 joined #gluster
12:48 aliguori joined #gluster
12:48 Norky http://www.gluster.org/pipermail/glu​ster-users/2012-October/034438.html for the gen about that XFS option
12:48 glusterbot <http://goo.gl/jklq6> (at www.gluster.org)
12:50 Norky this is not the same as the ext4 issue you're currently seeing. Insufficient total inodes is not the same thing as inodes too small to fit all the xattrs :)
12:50 Norky just to save any confusion
12:53 kke how do i even stop the gluster on one node
12:53 kke i did service glusterfs-server stop but there's still a lot of gluster processes
12:58 Norky .(gluster processes)
12:59 Norky .o(gluster processes)
12:59 * Norky has forgotten how to use the info bot
13:01 mohankumar__ joined #gluster
13:02 dewey_ joined #gluster
13:04 Norky you can kill them off, but I'm not perfectly sure how safe that is
13:04 kke or should i add a new brick and somehow use gluster to move the crap over to the new brick?
13:04 Norky I *think* anything like a self-heal process will be restarted, and if they're for bricks which you intend to remove anyway, no matter
13:05 kke from the bad ones
13:05 Norky http://www.mail-archive.com/gluste​r-users@gluster.org/msg11393.html has some useful info on the relationship between the init script and gluster porcesses
13:05 glusterbot <http://goo.gl/OmXQC> (at www.mail-archive.com)
13:05 kke and how does that work when they're mirrored?
13:05 kke or replicated
13:06 kke so i would need to move everything from storage1.vms:/export/data_2 to storage1.vms:/export/new_data_2  and same on storage2
13:07 Norky yes
13:08 Norky I believe you coudl do it piecemeal, kill gluster on one server, move data to a new XFS FS, unmount the ext4 and mount the XFS in its place, then restart gluster
13:08 kke but could i do it using gluster console?
13:08 kke some evacuate thing
13:09 Norky certainly that has worked for me, moving from a small birck to a larger, not tried between different FSes
13:09 kke there must be something like that for situations where you know the disk is failing
13:10 vpshastry joined #gluster
13:10 Norky this woudl be done from the shell, not particularly with the gluster coammdn, because of what you'd be doing is the parts *underneath* gluster
13:10 Supermathie left #gluster
13:10 kke there's something like: volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start|pause|abort|status|commit}
13:11 Norky that's also an option
13:11 Norky leave gluster running, and use its replace-brick feature
13:11 kke yeah
13:12 kke let's see how that works
13:12 Norky if this is replicated, then you have the option of letting gluster replicate files as appropriate to a new empty brick
13:12 Norky otherwise you'd need to copy the files from the old brick to the new
13:12 Norky that will likely be faster anyway
13:13 rob__ joined #gluster
13:13 kke but then i would have to take the volume offline and reads wouldn't work either
13:13 kke now they can read, not write
13:14 theron joined #gluster
13:14 hybrid5123 joined #gluster
13:15 Norky there will almost certainly be some self-heal needed afterwards, given that Gluster ran into an E_NOSPACE
13:15 Norky err, is the volume replicated?
13:16 Norky if it is replicated, then you take a single brick offline while the volume is still up
13:16 Norky clients will likely see some delay while the try to contact the 'missing' brick, but it shoudl still work
13:18 rob__ joined #gluster
13:18 Norky in fact, even if it is not replicated, merely distributed, you can take a brick down. Clients will then see half of your files
13:20 kke how do i take a brick offline? remove it?
13:23 thommy_ka joined #gluster
13:24 Norky actually, thinking about it, the best (fastest, safest) way is probably rsync -aX /oldbrick/ /newbrick  while everything is up, then do a replace-brick
13:24 kke maybe i just try to unmount it and mount it elsewhere
13:24 kke ok
13:24 kke replace brick doesn't want to co-operate with me
13:24 thommy joined #gluster
13:26 kke i tried to start it on storage2 by doing: volume replace-brick m1_data storage2.vms:/export/data_2 storage2.vms:/export/new_data2 start   it said "started", then i tried status, it said unknown, i tried start again and it says "replace-brick failed on storage1"
13:26 kke i tried to start it on storage1 and it fails with "replace-brick start failed" with no explanation
13:27 kke and yes, that is replicated
13:29 thommy joined #gluster
13:30 kke http://blog.adslweb.net/serendipity/article/2​94/Gluster-replace-a-brick-that-is-replicated that dude says it should work just like that
13:30 glusterbot <http://goo.gl/b7YXx> (at blog.adslweb.net)
13:34 kke that rsync is kind of slow
13:35 kke i don't understand what it's doing anyways, if i watch df -h it looks like disk used goes up, then it goes some way back again
13:35 kke two steps forward, one step back
13:35 joelwallis joined #gluster
13:35 Norky rsync will normally create a temporary file is it copies, then on completion move the temporary file to the real name
13:36 kke ok
13:38 kke could've been faster to do it over the network
13:39 kke 0.7GB of 80GB done
13:40 lh joined #gluster
13:40 lh joined #gluster
13:42 Norky hmm, feel free to try it a different way on the next one, but any slowness in 'direct; access is only likely to be compounded by the additional latency of network access
13:44 jruggiero joined #gluster
13:44 yinyin joined #gluster
13:45 rob__ joined #gluster
13:48 jruggiero joined #gluster
13:50 kke i got 10gbit network between the machines so it could've been faster to read on one node and write on another
13:50 kke instead of read+write on the same node
13:55 meunierd1 joined #gluster
13:56 portante joined #gluster
14:02 failshell joined #gluster
14:03 failshell what ports are used for geo replication?
14:04 rob__ joined #gluster
14:05 wushudoin joined #gluster
14:11 tziOm joined #gluster
14:12 meunierd1 Is there a typical way to let an unprivileged user write to a gluster volume?
14:13 Norky same with any mounted filesystem
14:13 failshell in geo replication, can a master be a removed volume?
14:14 failshell id like to pull the data
14:14 Norky create one or more directories within it and make them (group) writable
14:18 Norky you coudl even set the mode of the 'root' of the volume mount point
14:18 Norky sorry failshell, I know precious little about geo-replication
14:18 failshell only one way to find out: trying
14:24 jbrooks joined #gluster
14:32 RicardoSSP joined #gluster
14:32 RicardoSSP joined #gluster
14:33 kke hmm should the other brick in replace-brick be a brick that has been added or will it add it?
14:34 bugs_ joined #gluster
14:34 Norky the brick should not be part of any volume
14:34 kke ok
14:35 Norky oh, I've just had a thought, that rsync switch I told you might cause an (easily fixable) problem
14:35 kke so what do i do if i want to move stuff from a bad brick to the free space of another already added brick?
14:35 Norky setfattr -x trusted.glusterfs.volume-id $brick_path
14:35 Norky setfattr -x trusted.gfid $brick_path
14:35 Norky rm -rf $brick_path/.glusterfs
14:36 Norky ahh, essentially shrinking the volume by removing a brick?
14:36 kke essentially reducing the number of bricks by combining them
14:37 Norky combining?
14:37 kke because now i have one huge brick and i'd like to get rid of the small ones (as they have the inode problem)
14:39 45PAAJGPT joined #gluster
14:40 Norky okay, if your volume is currently running with the many small and one large bricks online, you want to "shrink"
14:40 Norky what kind of volume is this? (ssory, I'm sure you said, my memory is failing me)
14:41 Norky if it is replicated, you'll need to remove bricks in 2s
14:41 Norky mind you, if it is replciated, you will need another large brick
14:42 Norky sorry, I didn't realise you were substantial;ly changing brick sizes or my advice might have been different
14:47 kke i have been adding 80gb volumes as the free space on the volume has started to get low, now i have 6 * 80gb volumes (/export/data /export/data_2 ...) and they're replicated on another host which has identical layout, now i've added a 300gb volume and wish to move stuff from the old 80gb ext4 bricks to the 300gb xfs brick, then remove the 80gb bricks that have been made obsolete and create a new XFS volume in the freed space, then move stuff from the ...
14:47 kke ... remaining 80gb ext4's (6 * 80 won't fit on the 300gb volume obviously, but there isn't more space available for this operation)
14:47 kke volumes = bricks
14:47 kke on the same host
14:48 kke i'm up to /dev/vdk1 on the hosts :)
14:48 kke soooooo
14:49 vpshastry left #gluster
14:49 kke actually what i should have done is add the new 300gb xfs as brick, shrink the 100% inode using 80gb ext4, copy the data to the mounted glusterfs so that it replicates
14:49 JonnyNomad joined #gluster
14:50 kke right?
14:51 nueces joined #gluster
14:51 Norky I think "gluster vol replace-brick 80GBbrick 300GBbrick start" would have been best
14:51 Norky then the same on the other machine
14:52 Norky then shrink the volume a pair of 80GB bricks at a time
14:52 kke yeah but it didn't work
14:52 kke i tried that first
14:52 kke don't know why
14:52 Norky what failed exactly?
14:52 kke 16:24 < kke> replace brick doesn't want to co-operate with me
14:52 kke 16:26 < kke> i tried to start it on storage2 by doing: volume replace-brick m1_data storage2.vms:/export/data_2 storage2.vms:/export/new_data2 start   it said "started", then i tried status, it said unknown, i tried start again and it says "replace-brick failed on storage1"
14:52 kke 16:26 < kke> i tried to start it on storage1 and it fails with "replace-brick start failed" with no explanation
14:53 JoeJulian 3.3.1?
14:53 kke 3.2.1
14:53 JoeJulian Ah, that's why.
14:53 Norky ahah
14:53 kke so i'm screwed?
14:54 kke maybe i'll go with the shrink plan
14:54 JoeJulian You could upgrade to at least 3.2.7, though I would rather use 3.3 if I were trying to do that.
14:54 kke that's the newest available through apt-get
14:54 tqrst- joined #gluster
14:55 JoeJulian 3.2.1 has some critical bugs that'll bite you sooner or later.
14:55 JoeJulian @ppa
14:55 glusterbot JoeJulian: The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY -- and 3.4 packages are here: http://goo.gl/u33hy
14:55 failshell JoeJulian: maybe you know that. with geo replication, can the master be a remote volume?
14:55 kke will the upgrade take centuries? will old clients work?
14:55 JoeJulian failshell: Yes it can.
14:55 failshell JoeJulian: hmmm
14:56 JoeJulian kke: Upgrade takes moments, but the clients will have to be upgraded too. :(
14:56 kke and i'm on ubuntu oneiric so those ppa's don't have anything for me
14:56 kke yes, should upgrade to lts
14:56 failshell JoeJulian: when i run gluster  volume geo-replication remote:volume local:volume start, it prints the usage msg
14:56 JoeJulian failshell: Oh... just re-read that...
14:57 failshell *drum rolls*
14:57 JoeJulian failshell: You'd have to define it via the remote gluster command.
14:58 JoeJulian I wonder if the remote server switch to gluster would work...
14:58 kke i think i'll do the shrink thing
14:58 failshell was hoping to pull data from the remote site. now i have to wait for telecom to open the port in the other direction
14:58 failshell its only using SSH for that right?
14:59 JoeJulian gluster --remote-host=$remote
14:59 JoeJulian But no, that uses 24007
15:00 failshell why do you need to add SSH keys then?
15:02 JoeJulian I mean it's 24007 for the " gluster --remote-host=$remote" method of remote administration. If the target is a glusterfs volume, I don't think you need the tunnel. It'll mount the volume and sync it locally.
15:02 JoeJulian I think
15:05 kke hmm
15:05 45PAAJG00 joined #gluster
15:05 kke i ran remove-brick, now it says replace-brick is in progress
15:06 tqrst- does gluster{d,fs,fsd,shd,...} store state information anywhere other than in "working-directory" (/var/lib/glusterd)? I've been having long-standing stability issues with gluster, and suspect it might have something to do with our shared root setup. I'm just trying to see if all that needs to be in /etc/statetab really is in there.
15:06 daMaestro joined #gluster
15:07 kke so does remove-brick start some magical hidden replace-brick thing
15:07 saurabh joined #gluster
15:08 Norky both operations will (assuming you specify the "start" option) trigger a redistribution of files as appropriate
15:08 kke neat
15:09 Norky leaving out the "start" option does it instantaneously,a nd leaves the files behind - you'll need to manually fecth them form the old brick in that case
15:09 Norky at least under 3.3
15:10 Norky I dont' know about 3.2, I've only started using Gluster recently
15:10 Norky don't*
15:11 kke i have no idea where they're going
15:11 kke at least something is appearing on the new brick
15:12 kke looks like new data, not the old stuff
15:12 kke but obviously there's some replace going on
15:14 c4 joined #gluster
15:14 rob__ joined #gluster
15:17 jthorne joined #gluster
15:17 c3 joined #gluster
15:18 kke luster volume replace-brick m1_data storage1.vms:/export/data_2 storage1.vms:/export/new_data2 status
15:18 kke brick: storage1.vms:/export/data_2 does not exist in volume: m1_data
15:18 kke gluster volume rebalance m1_data fix-layout start
15:18 kke Replace brick is in progress on volume m1_data
15:18 lpabon joined #gluster
15:22 kke gotta take a break and see what's left of my data in about an hour
15:23 c4 joined #gluster
15:25 spider_fingers left #gluster
15:28 Technicool joined #gluster
15:35 kaptk2 joined #gluster
15:37 semiosis hagarth__: yes i just set up the 3.3qa PPA on friday: https://launchpad.net/~semiosis/​+archive/ubuntu-glusterfs-3.3qa
15:37 glusterbot <http://goo.gl/5fnXN> (at launchpad.net)
15:41 manik joined #gluster
15:42 vpshastry joined #gluster
15:42 vpshastry left #gluster
15:44 bennyturns joined #gluster
15:47 devoid joined #gluster
15:56 c3 joined #gluster
16:02 nueces joined #gluster
16:03 saurabh joined #gluster
16:04 nueces joined #gluster
16:06 jag3773 joined #gluster
16:15 vpshastry joined #gluster
16:18 failshell after starting geo replication, i keep getting that error: https://gist.github.com/fai​lshell/980432792a1ba07120a1
16:18 glusterbot <http://goo.gl/H5RVy> (at gist.github.com)
16:18 failshell and the status is at faulty
16:19 failshell passwordless SSH works
16:20 ninkotech joined #gluster
16:20 ninkotech_ joined #gluster
16:28 failshell do you need to peer probe bricks that do geo-replication?
16:28 hjmangalam1 joined #gluster
16:35 lh joined #gluster
16:37 Mo_ joined #gluster
16:41 _pol joined #gluster
16:41 semiosis ~glossary | failshell
16:41 glusterbot failshell: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
16:41 failshell that does not explain anything
16:42 semiosis your question doesnt make sense
16:42 semiosis you never probe bricks, you probe servers
16:42 semiosis bricks dont do geo-rep, servers do
16:42 failshell i meant servers
16:47 nueces_ joined #gluster
16:53 nueces joined #gluster
17:02 manik joined #gluster
17:02 semiosis joined #gluster
17:09 _pol joined #gluster
17:10 _pol joined #gluster
17:23 anands joined #gluster
17:27 hagarth joined #gluster
17:28 nueces joined #gluster
17:30 lpabon joined #gluster
17:36 vpshastry left #gluster
17:40 tziOm joined #gluster
17:45 hagarth joined #gluster
17:48 thomaslee joined #gluster
17:49 zaitcev joined #gluster
18:03 hagarth joined #gluster
18:06 Technicool joined #gluster
18:12 krishna joined #gluster
18:13 c3 joined #gluster
18:18 krishna_ joined #gluster
18:19 hchiramm_ joined #gluster
18:29 krishna__ joined #gluster
18:33 Chocobo joined #gluster
18:33 Chocobo joined #gluster
18:38 failshell im trying to get georep to work. i keep getting that error on the master: https://gist.github.com/fai​lshell/8ac49219cca6abe26123
18:38 glusterbot <http://goo.gl/zG7Zc> (at gist.github.com)
18:39 failshell and my georep is faulty
18:48 kspaans_ joined #gluster
18:56 glusterbot New news from newglusterbugs: [Bug 970224] Under heavy load, Grid Engine array jobs fail; write permission <http://goo.gl/LhxYx>
19:00 _pol joined #gluster
19:02 kspaans joined #gluster
19:08 lbalbalba joined #gluster
19:09 * Kripto_ is seeing some odd issues with GlusterFS (3.3.1-1.el6.x86_64) and samba (3.5.10-125.el6.x86_64) specifically with files being written to a smb share that points to GFS mount, the files aren't closing their filehandles. Resulting in "sparse" files. The files are 150k gifs/jpegs. Anyone seen similar issues?
19:11 JoeJulian Are sparse files a bad thing?
19:12 Kripto_ JoeJulian: Yes, they are not viewable in this state.. if you "vi" them you see "noeol"
19:12 Kripto_ which means that the files weren't closed and therefore never got the final descriptor bit.
19:14 Kripto_ also, note that when we set the smb target to a non glusterfs mount, it works as expected.
19:14 JoeJulian Ok, so the question is, who's not closing their fd? The samba client (Windows?) or samba?
19:15 JoeJulian Perhaps a wireshark analysis and/or an strace would tell you something.
19:16 Kripto_ JoeJulian: I suspect that smb is not the issue, because when we point to a local filesystem we don't see the issue..
19:24 Kripto joined #gluster
19:27 ccha2 what mean fd ? file descripstor ?
19:28 Kripto yes
19:40 andreask joined #gluster
20:03 ThatGraemeGuy joined #gluster
20:03 rb2k joined #gluster
20:07 meunierd1 There's no way to increase replica count, is there?
20:07 jbourke joined #gluster
20:07 meunierd1 If I have a count of 2, and want to add a third server?
20:08 hchiramm_ joined #gluster
20:08 JoeJulian add-brick replica 3 {new brick list}
20:10 meunierd1 to an existing volume?
20:10 JoeJulian Assuming you're using version 3.3
20:11 meunierd1 Ah, I think I might still be on 3.2
20:11 meunierd1 I'll upgrade and attempt that.
20:12 rb2k joined #gluster
20:13 jbourke I'm an old 3.1 user, and am setting up a 3.3 cluster for my new company.  I've got a situation where one peer is connected, and one is not.  Both show 'Peer in Cluster", but one shows 'Connected', the other 'Disconnected'.  I've checked tcpdump to see packets back and forth and am using IPs, not DNS.  Anyone have a thought for the next thing to poke?
20:14 jbourke I've been searching chat logs and google for hours now trying to figure out what is up when both can see each other and peer, but not actually stay connected.
20:15 semiosis iptables
20:15 JoeJulian @ports
20:15 glusterbot JoeJulian: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
20:15 JoeJulian jbourke: Also try restarting glusterd on all your servers. I've seen that clear things like that up too.
20:25 jbourke I've been restarting consistantly watching tcpdump for clues.  They seem to see each other.  They did peer.  But, the secondary can't stay connected to the primary in order to make a volume.
20:25 jbourke These are fresh servers, I will cordon them off and clear any iptables and see what happens.  Internal traffic is unregulated, but who knows.
20:26 JoeJulian selinux?
20:26 hchiramm_ joined #gluster
20:26 jbourke Ubuntu 12.04.  Nathin special.
20:27 jbourke Oh, jeez.  Ok, coffee time.
20:27 JoeJulian hehe
20:27 jbourke I have not checked my security settings yet.  Good point, I may have something up I don't expect.
20:29 semiosis jbourke: on the machine you call 'secondary' is the glusterd process dying?
20:31 _pol joined #gluster
20:31 meunierd1 Is ppa:semiosis/ubuntu-glusterfs-3.3 the official ppa? I found another one that's apparently deprecated.
20:31 _pol joined #gluster
20:32 semiosis meunierd1: yes that's the right one
20:32 semiosis ,,(ppa)
20:32 glusterbot The official glusterfs 3.3 packages for Ubuntu are available here: http://goo.gl/7ZTNY -- and 3.4 packages are here: http://goo.gl/u33hy
20:33 semiosis i just put up a 3.3qa PPA last week...
20:33 semiosis @forget ppa
20:33 glusterbot semiosis: The operation succeeded.
20:34 semiosis @learn ppa as The official glusterfs 3.3 packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.3 QA: https://launchpad.net/~semiosis/​+archive/ubuntu-glusterfs-3.3qa -- and 3.4 QA: http://goo.gl/u33hy
20:34 glusterbot semiosis: The operation succeeded.
20:34 semiosis @ppa
20:34 glusterbot semiosis: The official glusterfs 3.3 packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.3 QA: http://goo.gl/5fnXN -- and 3.4 QA: http://goo.gl/u33hy
20:34 meunierd1 Is 3.3 -> 3.4 an easier upgrade path than 3.2 -> 3.3 or should I just skip ahead?
20:35 JoeJulian It's /supposed/ to be an easier upgrade.
20:36 JoeJulian The 3.4 rpc /should/ talk to 3.3. I'm not sure how tested that is though.
20:40 meunierd1 And the difference between the old ppa and the new one is that it's provided to overwrite the existing glusterfs-server package rather than create a new glusterd one?
20:40 JoeJulian semiosis: ^
20:41 semiosis yes
20:41 semiosis s/yes/affirmative/
20:41 glusterbot What semiosis meant to say was: meunierd1: affirmative that's the right one
20:41 semiosis sure
20:41 JoeJulian lol
20:42 meunierd1 Awesome, thanks
20:42 JoeJulian That actually worked out okay. Had me scratching my head for a second though.
20:42 JoeJulian glusterbot's become self-aware...
20:43 semiosis binary solo
20:44 rb2k joined #gluster
21:02 _pol joined #gluster
21:03 _pol_ joined #gluster
21:04 _pol joined #gluster
21:16 _pol joined #gluster
21:22 cfeller joined #gluster
21:22 budric joined #gluster
21:22 _pol_ joined #gluster
21:24 budric Hi, I'm having trouble getting quick start to work in ubuntu server 13.04.  Gluster 3.2.7 doesn't mount on a client machine - transport endpoint is not connected.  There should be no network issues as I can telnet to one of the servers on port 24009.  I tried upgrading to 3.3 from PPA, now the gluster server fails with gf_xdr_from_cli_defrag_vol_req.  I'm just trying to setup a simple 2 server replication.
21:25 semiosis 3.2 and 3.3 are not interoperable
21:25 semiosis i suggest you start over with the latest 3.3 packages
21:26 semiosis or if you're feeling adventurous try the 3.3.2 or 3.4.0 QA releases, also in PPAs
21:26 _pol joined #gluster
21:26 budric Ok, but when I upgrade one of the servers and try gluster --version it fails with undefined symbol: gf_xdr...
21:26 semiosis hmmm
21:27 semiosis oh and i havent uploaded raring packages for 3.3.2 or 3.4.0 yet (no one asked)
21:27 semiosis if i upload those now for you it looks like it could be a few hours before they get built by launchpad :/
21:27 budric oh that's cool.  I wonder what was installed...
21:28 semiosis 3.3.1 is the latest stable glusterfs release, i do have that for raring in the PPA
21:28 semiosis ,,(ppa)
21:28 glusterbot The official glusterfs 3.3 packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.3 QA: http://goo.gl/5fnXN -- and 3.4 QA: http://goo.gl/u33hy
21:28 semiosis 1st link has a raring package
21:29 budric I used this ppa from this: http://download.gluster.org/pub/gluster/​glusterfs/3.3/3.3.1/Ubuntu/Ubuntu.README .  dpkg --list | grep gluster shows 3.3.1-ubuntu1~raring9
21:29 glusterbot <http://goo.gl/AvrSq> (at download.gluster.org)
21:29 budric should I reinstall from those links?
21:30 budric oh, looks like the same ppa
21:35 budric Sorry, look like my fault.  the common package installed was 3.3.1, the server package was still 3.2.7.  One server seems to be up
21:36 Technicool joined #gluster
21:36 semiosis great
21:36 semiosis wb Technicool
21:36 Technicool ty ty  ;-)
21:39 _pol joined #gluster
21:40 rb2k joined #gluster
21:44 _pol joined #gluster
21:49 _pol joined #gluster
21:58 _pol_ joined #gluster
22:00 _pol__ joined #gluster
22:00 budric I still can't mount it on the client.  Logs show a warning Transport endpoint is not connected peer (142.224.240.170:24007), followed by error regarding GlusterFS handshake.  But I can telnet to that ip and that port so the network should work.  Any suggestions what I can check?
22:04 rcoup joined #gluster
22:07 rcoup morning folks. Couple of questions :) During a remove-brick (rebalance), is it expected that clients will be continuously spitting out errors? http://dpaste.com/1210141/ The filesystem *seems* okay
22:07 glusterbot Title: dpaste: #1210141 (at dpaste.com)
22:07 rcoup There's no split-brain errors
22:07 JoeJulian budric: If you were running a redhat based distro, I'd immediately jump to selinux.
22:07 rcoup actually, there are  now :/
22:09 JoeJulian rcoup: Check, for me, and see of .glusterfs/00/00/00000000-0​000-0000-0000-000000000001 is a directory instead of a symlink on any of your bricks.
22:09 rcoup (http://dpaste.com/1210146/)
22:09 glusterbot Title: dpaste: #1210146 (at dpaste.com)
22:09 * rcoup looks
22:12 rcoup JoeJulian: hmmm. Looks like a storage brick on one host is dying/dead: IO errors.
22:13 rcoup #yayfun
22:14 ThatGraemeGuy joined #gluster
22:17 rcoup hmm, think the filesystem just needs remounting.
22:17 rcoup glusterfs stop on that node, fix the FS, then restart and self-heal will kick off, right?
22:18 rcoup should I stop the rebalance first?
22:19 clag_ joined #gluster
22:22 JoeJulian I would
22:35 jag3773 joined #gluster
22:38 plarsen joined #gluster
22:42 rcoup seems happier. Well, once it's rebalanced everything again :/ Thanks JoeJulian
22:51 budric I mounted a volume on one of the clustered servers that worked and ran sequential read benchmark.  I'm getting about 10 Mb/second or equivalent to network speed.  So I take it every read operation needs to synchronize between all servers?  GlusterFS doesn't return local copies for faster reads?
22:54 JordanHackworth joined #gluster
22:55 JoeJulian replicate serves from "first to respond". In a scaled cluster environment, this provides the best performance.
22:56 JordanHackworth joined #gluster
22:56 JoeJulian generally
22:56 JoeJulian I think 3.4 is supposed to have some method for overriding that behavior though. I haven't had a chance to look yet.
22:57 rcoup JoeJulian: if you know, where's the code for that? We've seen (in 2-node replicas) load often going 100% to one node, despite being further away (+5ms) from the idle node.
22:57 rcoup At other times it'll be even
22:57 JoeJulian should be in xlators/afr
22:58 rcoup my suspicion is a bug, which appears a bit more frequently in 2-node scenarios and is smoothed over in clusters with more nodes.
22:58 rcoup will have a look, thanks
22:58 JoeJulian I wouldn't be entirely surprised. Perhaps first to respond is usually first to be asked...
22:59 budric does it matter how you mount it?  eg localhost:/gv0 vs fully.qualified.name:/gv0
22:59 JoeJulian no
22:59 JordanHackworth joined #gluster
22:59 JoeJulian @mount host
22:59 JoeJulian @mount server
22:59 glusterbot JoeJulian: (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
23:00 budric that page is not found, but I read the admin guide.  I know what you're referring to
23:01 JoeJulian hmm, may have to wayback that and make a wiki page...
23:02 JordanHackworth joined #gluster
23:02 _pol joined #gluster
23:03 _pol joined #gluster
23:05 _pol joined #gluster
23:05 _pol joined #gluster
23:11 _pol joined #gluster
23:12 _pol joined #gluster
23:13 JordanHackworth joined #gluster
23:25 portante joined #gluster
23:25 duerF joined #gluster
23:26 rob__ joined #gluster
23:27 rcoup JoeJulian: looks like someone was already looking at this: http://review.gluster.com/#/c/2926/
23:27 glusterbot Title: Gerrit Code Review (at review.gluster.com)
23:27 rcoup would that have made it into 3.3.x?
23:28 rcoup hmm, apparently not
23:37 StarBeast joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary