Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2013-12-06

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 sarkis joined #gluster
00:05 TrDS hi... how can the hostname of a peer (and its bricks) be renamed in gluster 3.3?
00:14 dbruhn joined #gluster
00:15 semiosis TrDS: you might have to take the servers offline & do a search & replace on the config files
00:16 semiosis last time i looked into that (over a year ago) there wasnt a convenient way to do it with the gluster command
00:16 TrDS ok
00:16 TrDS thx
00:16 semiosis s/there wasnt/i never found/
00:16 glusterbot What semiosis meant to say was: [histsearch i never found]
00:16 TrDS someone kinda wrote this on the mailing list
00:16 semiosis that bot is in bad shape
00:16 JoeJulian eww
00:17 semiosis JoeJulian: also note the shortlinks are all broken
00:17 semiosis file a bug for example
00:17 glusterbot http://goo.gl/p7D4Fh
00:17 TrDS but this seems like you really want to have integrated
00:17 JoeJulian grr
00:17 semiosis TrDS: file a bug requesting the feature i guess
00:17 glusterbot http://goo.gl/p7D4Fh
00:18 semiosis here's the real link... https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
00:18 glusterbot <http://goo.gl/p7D4Fh> (at bugzilla.redhat.com)
00:18 TrDS thx
00:19 semiosis TrDS: fwiw my advice to avoid having to change a hostname in the first place is to set up your servers with dedicated dns names, gluster1.my.domain.net for example
00:19 semiosis then cname/a that onto the actual host serving the gluster1 role
00:19 semiosis actually my servers are named things like front-left
00:20 TrDS semiosis: that's want i indend to do now, but when i started testing gluster, i just used the "normal" dns names
00:20 semiosis ah
00:25 mattapp__ joined #gluster
00:30 harish joined #gluster
00:32 verdurin joined #gluster
00:33 mattapp__ joined #gluster
00:36 davidbierce joined #gluster
00:37 mattappe_ joined #gluster
00:45 TrDS is there a way to reference another ticket in the redhat bug tracker? it's not a "depends on", more a "related with"
00:50 hchiramm_ joined #gluster
00:51 mattappe_ joined #gluster
00:57 TrDS this bugzilla (instance?) does not even have a preview function, i don't like it ^^
00:58 TrDS can you edit your own entries?
00:59 _pol joined #gluster
01:04 mkzero joined #gluster
01:18 mattapp__ joined #gluster
01:26 glusterbot New news from newglusterbugs: [Bug 1038866] [FEAT] command to rename peer hostname <http://goo.gl/3x0Wlf>
01:31 harish joined #gluster
01:56 TrDS left #gluster
02:15 rjoseph joined #gluster
02:22 gtobon joined #gluster
02:26 bharata-rao joined #gluster
02:36 shubhendu joined #gluster
02:36 sarkis joined #gluster
02:48 johnmark reminder: glusterfest testing weekend starting Real Soon Now: http://www.gluster.org/2013/12/it​s-a-glusterfest-testing-weekend/
02:48 glusterbot <http://goo.gl/Rdy6Zg> (at www.gluster.org)
03:01 harish joined #gluster
03:02 gtobon joined #gluster
03:15 mattapp__ joined #gluster
03:16 kshlm joined #gluster
03:21 sgowda joined #gluster
03:29 X3NQ joined #gluster
03:32 kanagaraj joined #gluster
03:32 RameshN joined #gluster
03:38 _pol joined #gluster
03:38 itisravi joined #gluster
03:40 Alex Anyone know if you need to run fix-layout and then migrate-data after adding a new brick, or whether you can just run rebalance?
03:40 Alex http://gluster.org/community/documentation​/index.php/Gluster_3.2:_Rebalancing_Volume​_to_Fix_Layout_and_Migrate_Existing_Data indicates you can just run rebalance, but I'm not 100% certain
03:40 glusterbot <http://goo.gl/8tb5W8> (at gluster.org)
03:47 jag3773 joined #gluster
03:47 bala joined #gluster
03:57 atrius joined #gluster
03:59 mohankumar joined #gluster
04:06 shruti joined #gluster
04:07 dusmant joined #gluster
04:14 saurabh joined #gluster
04:17 nshaikh joined #gluster
04:21 ndarshan joined #gluster
04:30 mattappe_ joined #gluster
04:31 mattapp__ joined #gluster
04:35 gtobon joined #gluster
04:48 ppai joined #gluster
04:48 mistich joined #gluster
04:50 mistich anyone know what this error is [2013-12-06 04:45:44.432168] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: 17229fab-644b-4b1d-941e-f2f6fc529823, lock held by: 17229fab-644b-4b1d-941e-f2f6fc529823
04:51 mistich ok my cluster just got worse  E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so​.0(rpc_clnt_notify+0x13d) [0x33d400ec8d] (-->/usr/lib64/libgfrpc.so.0(rp​c_clnt_connection_cleanup+0xc3) [0x33d400e7f3] (-->/usr/lib64/libgfrpc.so.​0(saved_frames_destroy+0xe) [0x33d400e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(2)) called at 20
04:51 mistich now spitting out this over and over
04:52 shyam joined #gluster
04:57 spandit joined #gluster
04:58 meghanam joined #gluster
04:58 meghanam_ joined #gluster
05:13 JoeJulian mistich: Looks like glusterd got into some sort of loop trying to lock something. Restarting glusterd on whichever server has that uuid /should/ fix it.
05:13 JoeJulian Alex: A simple rebalance does both.
05:16 Alex Thanks JoeJulian. I shall now watch, and wait.
05:17 Alex What generally determines the performance of a rebalance? iowait is <10% according to top, network throughput is low hundreds kbit/s, and the CPU doesn't seem "very" busy. Just wondering if there's a "go faster" button :)
05:19 JoeJulian I've never looked for that. I do know that it runs at a lower priority than regular traffic.
05:19 mattapp__ joined #gluster
05:22 Alex Has been running for 2h, and has done around 100GB, so I guess I'll look into priorisiation. Thanks!
05:22 mistich JoeJulian restarted  and still cannot not do a volume status just hangs
05:24 sgowda joined #gluster
05:25 CheRi joined #gluster
05:28 mistich easier question can you create a Distributed Replicated with servers that have multiple bricks
05:31 Alex yeah, that's certainly a bit happier now. (Higher priority)
05:35 rastar joined #gluster
05:37 satheesh joined #gluster
05:38 JoeJulian mistich: yes. I do that myself. And if restarting one glusterd didn't fix it, have them all stopped simultaneously. What version are you running?
05:41 mistich glusterfs-3.4.1-3.el6.x86_64
05:42 mistich I have tried that too it will not even let me delete the volume either
05:43 mistich would like to save the data but at this point just test data
05:43 mistich [2013-12-06 05:43:15.083198] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x17x sent = 2013-12-06 05:33:05.017349. timeout = 600
05:43 mistich [2013-12-06 05:43:25.084436] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(1)) xid = 0x18x sent = 2013-12-06 05:33:24.675354. timeout = 600
05:44 hagarth joined #gluster
05:44 psharma joined #gluster
05:45 JoeJulian mistich: That's 5 minutes. That would seem that glusterd's been running for at least that long. On each server, "service glusterd stop". Make sure it's really stopped. It's the only one named glusterd. glusterfsd and glusterfs are ok. Once you're sure it's stopped on every server, then you can "service glusterd start" again.
05:45 JoeJulian @processes
05:45 glusterbot JoeJulian: The GlusterFS core uses three process names: glusterd (management daemon, one per server); glusterfsd (brick export daemon, one per brick); glusterfs (FUSE client, one per client mount point; also NFS daemon, one per server). There are also two auxiliary processes: gsyncd (for geo-replication) and glustershd (for automatic self-heal). See http://goo.gl/F6jqx for more information.
05:46 JoeJulian @unload shorturl
05:46 glusterbot JoeJulian: Error: There was no plugin shorturl.
05:49 mistich JoeJulian still cannot delete volume
05:49 mistich just sitting there
05:50 mistich any suggestion on how to remove the volume
05:51 aravindavk joined #gluster
05:53 davinder joined #gluster
05:54 dusmant joined #gluster
06:03 JoeJulian mistich: Gah! That doesn't make any sense! To brute-force it, though, if you don't care about the volume, stop all glusterd and glusterfsd and rm -r /var/lib/glusterd/vols/*
06:04 JoeJulian I saw that happen with 3.4.0, but not with 3.4.1... :(
06:05 mistich yeah in the process thanks
06:05 mistich if happens again will submit a bug
06:13 mistich joejulian one more question since im rebuilding I have 6 drives in raid1 pairs would i get better performance if I removed the raid 1 and had 6 bricks per server rather than 3 bricks per server I have a total of 9 server 10gig network with ssd drives all files are around 4 mb
06:14 satheesh1 joined #gluster
06:14 mistich since its going to be a Distributed-Replicate raid 1 not needed for redundancy
06:15 JoeJulian mistich: Depends on the load. If you have one client reading whole files, raid will be faster. If you have thousands of clients reading thousands of different files, then 6 bricks will be better.
06:15 JoeJulian s/will/should/
06:15 glusterbot What JoeJulian meant to say was: [histsearch should]
06:15 JoeJulian @meh
06:15 glusterbot JoeJulian: I'm not happy about it either
06:15 JoeJulian @histsearch should
06:16 mistich 3 process per server and 20+ server accessing different files
06:16 mistich will break it up into 6 thanks
06:18 krypto joined #gluster
06:19 satheesh joined #gluster
06:27 bulde joined #gluster
06:28 geewiz joined #gluster
06:46 lalatenduM joined #gluster
06:49 shubhendu joined #gluster
06:51 itisravi_ joined #gluster
06:57 glusterbot New news from newglusterbugs: [Bug 1024181] Unicode filenames cause directory listing interactions to hang/loop <https://bugzilla.redhat.co​m/show_bug.cgi?id=1024181>
07:00 anands joined #gluster
07:04 raghu joined #gluster
07:05 dylan_ joined #gluster
07:17 shubhendu joined #gluster
07:19 social JoeJulian: what do you suggest if I lost all data from one brick, I have node with same name spinned up but it throws -posix: Directory '/mnt/gluster/brickdir' doesn't exist, exiting.
07:20 social I should probably recreate brick dir and apply http://joejulian.name/blog/repl​acing-a-brick-on-glusterfs-340/ to get it healed?
07:20 glusterbot Title: Replacing a brick on GlusterFS 3.4.0 (at joejulian.name)
07:26 satheesh1 joined #gluster
07:26 mistich does gluster when you create file use the 1st brick then when it is full moves to the 2nd brick etc.. or is it suppose to put files randomly across all bricks
07:27 mistich I'm seeing the only io on the 1st brick for now
07:27 glusterbot New news from newglusterbugs: [Bug 1038924] glfs_fini in libgfapi closes stderr if stderr is used as logfile <https://bugzilla.redhat.co​m/show_bug.cgi?id=1038924>
07:32 jtux joined #gluster
07:40 mbukatov joined #gluster
07:46 davidbierce joined #gluster
07:50 getup- joined #gluster
07:54 Alex Just want to confirm a design decision - where I have two gluster servers, and 6 bricks per server, if I'm using distribute-replicate, is there a way to guarantee that in the event of the loss of one server, the files will all be present on the other, or do I need to configure something further to allow that?
07:54 ctria joined #gluster
07:54 Alex (he says, reading http://joejulian.name/blog/glust​erfs-replication-dos-and-donts/ :))
07:54 glusterbot Title: GlusterFS replication dos and donts (at joejulian.name)
07:57 samppah Alex: you have to present brick as replication pairs to ensure that
07:57 samppah ie. gluster vol create repVol replica 2 server1:/gluster1 server2:/gluster1 server1:/gluster2 server2:/gluster2
07:58 samppah it would replicate files between gluster1 directory on server1 and server2
07:58 Alex samppah: Ah, understood - I guess my concern there is that I don't have a clear picture of how I could then go from a single node deployment to a multi node deployment
07:58 samppah Alex: do you have setup running now or still planning it?
07:58 Alex samppah: I'm planning it, more-or-less :)
07:59 samppah okay, if you had one node setup and created it with gluster vol create voLName server1:/gluster1
07:59 samppah then you can start replicating it with gluster vol add-brick volName replica 2 server2:/gluster1
08:00 samppah after that you have to add at least to bricks at once and it will replicate data between those
08:00 samppah so gluster vol vol add-brick volname replica 2 server1:/gluster2 server2:/gluster2
08:01 tziOm joined #gluster
08:01 Alex Ah, right, understood, let me just test that to see if it fits what I'm thinking, as I realisae realise I may not have explained myself properly. Thanks samppah! :)
08:01 samppah Alex: no problem. feel free to ask if you anything comes up :)
08:02 Dga joined #gluster
08:05 eseyman joined #gluster
08:06 Alex samppah: ah, okay, so I think then I need to reconsider my approach for brick creation. At the moment I've just got a straight mapping (essentially) of HDs -> bricks, but it doesn't seem like there's a way I could handle adding two bricks at once? So if I have... https://gist.github.com/1177fd6603a1dbbb55dd - I couldn't then just say 'okay, now make ca2.gl:/data0/gluster a pair for ca1.gl:/data0/gluster', right?
08:06 glusterbot Title: gist:1177fd6603a1dbbb55dd (at gist.github.com)
08:07 samppah Alex: unfortunately not.. i'm not sure if this could be possible with 3.5 that is coming up, http://www.gluster.org/community/documentat​ion/index.php/Features/AFR_CLI_enhancements
08:07 glusterbot Title: Features/AFR CLI enhancements - GlusterDocumentation (at www.gluster.org)
08:08 samppah but iirc it should work like gluster vol add-brick shared replica 6 ca2.gl:/data0/gluster ca2.gl:/data1/gluster ca2.gl:/data2/gluster ca2.gl:/data3/gluster ca2.gl:/data4/gluster ca2.gl:/data5/gluster
08:10 samppah i have never tried that myself with that many bricks but it should look at the order and match up with existing setup
08:10 Alex samppah: ah, that's a real shame - so we're very much limited to either LVM all of the disks together and then treat that as a brick, or other RAID, I guess. (but then of course, you don't have RAID in the LVM setup so if you lost a disk you wouldn't have a way to repair so actually, ignore this :)
08:10 Alex *nod* - looks like it throws Incorrect number of bricks (6) supplied for replica count (6)., which kind of makes sense - as it's actually 2 replicas per brick
08:10 samppah oops
08:10 samppah sorry
08:10 samppah my mistake
08:11 samppah it's replica 2 not 6 :)
08:11 * Alex strokes chin
08:12 Alex Ahh, the only difference in this case is that when I added new bricks, I added it to a volume that wasn't created as a replica. Let me see if that had a positive difference
08:13 samppah you should be able to change replication level with add-brick too
08:13 samppah at least with glusterfs 3.4
08:14 Alex I think that worked, actually
08:14 Alex changing the replication level, that is
08:16 samppah i'll be away for a moment.. i have to take the dogs for a walk :)
08:16 Alex Np - just going to re-test, but I think that worked :)
08:21 mohankumar joined #gluster
08:26 mistich never-mind I forgot to mount bricks
08:26 _pol_ joined #gluster
08:27 ccha3 joined #gluster
08:28 JonnyNomad_ joined #gluster
08:29 Nuxr0 joined #gluster
08:29 darshan joined #gluster
08:30 ulimit_ joined #gluster
08:30 pravka_ joined #gluster
08:30 nixpanic_ joined #gluster
08:30 xymox joined #gluster
08:31 nixpanic_ joined #gluster
08:35 Remco joined #gluster
08:35 Remco joined #gluster
08:36 _Bryan_ joined #gluster
08:41 dusmant joined #gluster
08:49 _polto_ joined #gluster
08:49 _polto_ joined #gluster
09:02 shubhendu joined #gluster
09:06 satheesh1 joined #gluster
09:12 vimal joined #gluster
09:14 itisravi joined #gluster
09:17 sac`away joined #gluster
09:18 StarBeast joined #gluster
09:26 RedShift joined #gluster
09:39 TrDS joined #gluster
09:42 social hagarth: http://paste.fedoraproject.org/59489/32293813/ culprint of oomkill
09:42 glusterbot Title: #59489 Fedora Project Pastebin (at paste.fedoraproject.org)
09:45 mistich anyone know how to look at old bugs http://bugs.gluster.com/show_bug.cgi?id=3011
09:45 glusterbot Bug 3011: medium, medium, ---, dkl, CLOSED NOTABUG, Xnest dumps core
09:47 davidbierce joined #gluster
09:48 hagarth social: is that so?
09:48 social hagarth: it might not be memleak at all just some stray loop
09:51 tru_tru joined #gluster
09:51 nonsenso joined #gluster
09:51 abyss^ joined #gluster
09:52 hagarth social: i am wondering if there is a missing mem_put, let me check further
09:53 klaxa joined #gluster
09:53 darshan joined #gluster
09:58 social hagarth: https://bugzilla.redhat.co​m/attachment.cgi?id=833514 this is probably better for you
09:59 ProT-0-TypE joined #gluster
09:59 dusmant joined #gluster
10:01 satheesh1 joined #gluster
10:01 lkoranda joined #gluster
10:09 jasda joined #gluster
10:18 kshlm mistich: the old gluster bugs had aliases created for them in the redhat bugzilla as 'GLUSTER-<bug-id>" . So the link for your bug will be  https://bugzilla.redhat.com/a​ttachment.cgi?id=GLUSTER-3011
10:18 glusterbot Title: Invalid Attachment ID (at bugzilla.redhat.com)
10:18 ricky-ti1 joined #gluster
10:19 kshlm Oops, correct link is https://bugzilla.redhat.com/​show_bug.cgi?id=GLUSTER-3011
10:19 glusterbot Bug GLUSTER: could not be retrieved: InvalidBugId
10:20 samsamm joined #gluster
10:20 JonathanD joined #gluster
10:34 dusmant joined #gluster
10:39 mohankumar joined #gluster
10:44 CheRi joined #gluster
10:44 _pol joined #gluster
10:50 partner hmm now my new brick started to suck in lots of files and the brick log is giving lots of these: [2013-12-06 10:47:53.740674] E [posix.c:1755:posix_create] 0-dfs-posix: setting xattrs on /all-my-files.. failed (Operation not supported)
10:53 hagarth joined #gluster
10:53 ppai joined #gluster
10:54 partner previously it was MKNOD/UNLINK failures (permission denied) and i just can't find a reason for that, after all the gluster runs as root
10:55 partner whaat: [2013-12-06 10:55:20.882632] W [posix-helpers.c:701:posix_handle_pair] 0-dfs-posix: Extended attributes not supported (try remounting brick with 'user_xattr' flag)
10:56 lalatenduM partner, what is the filesystem you are using under Gluster?
10:57 partner xfs with 512 inode size
10:58 partner 3.3.2 on debian wheezy
10:58 mohankumar joined #gluster
10:58 lalatenduM partner, xfs with 512 inode size is fine, you did add brick and rebalance?
11:00 partner yeah i added brick some time ago, new server with 1 new brick, no issues there, its now sixth server on this distributed volume. doing fix-layout now for yet another week or two
11:00 partner maybe i have failed somewhere on the deployment but i have hard time figuring out where as all the commands were copypasted from our internal setup docs..
11:01 partner ie. 6 identical servers
11:02 partner hmm could this be caused by the client side? i think there's possibly 3.3.1 client while the servers are 3.3.2 but as no packages were not available i could not upgrade clients
11:02 lalatenduM partner, check gluster v info <vol> and gluster v status <vol> it might give your some more idea what is wrong
11:03 lalatenduM ideally we should update the clients too
11:03 partner nothing reports errors anywhere but in the brick logs
11:03 lalatenduM where do you see the error?
11:03 lalatenduM got it
11:03 lalatenduM only brick logs
11:04 lalatenduM you mean application is running fine? you dont see an error there
11:04 partner rebalance log just slow keeps saying its fixing this and that directory layout, not seeing errors
11:04 partner no, nothing visible on the client side
11:04 lalatenduM hmm
11:04 partner let me check the logs from there too but as for files they are storable and accessible just fine
11:05 partner i have nothing special for the fstab entries, very simple setup, 6 brick distributed
11:07 partner i see no errors on client side, just some warnings subvolumes are full.. well thats the reason for adding new bricks..
11:07 partner only major thing that has been done was upgrading from 3.3.1 to 3.3.2 earlier to get rid off the rebalance leaking file handlers
11:07 lalatenduM ok, I have never seen this error in log.   I will suggest you to send an email to gluster-users , somebody might know
11:08 lalatenduM partner, http://www.gluster.org/interact/mailinglists/
11:08 partner looks a bit like this one: https://bugzilla.redhat.com/show_bug.cgi?id=850873
11:08 glusterbot Bug 850873: low, unspecified, ---, amarts, CLOSED CURRENTRELEASE, glusterfsd tries to set extended attribute "umask" when files are created
11:09 lalatenduM looking
11:10 lalatenduM partner, yes, you are right
11:10 lalatenduM partner, this bug is fixed in  3.4
11:10 partner so it seems
11:11 partner possible that has been logged earlier too but i'm not exactly tailing the logs each day..
11:11 partner now i've been as there were issues getting files stored into new brick with "expected volume", it was just almost idling while 5 other nodes were doing huge loads
11:13 _amir joined #gluster
11:13 CheRi joined #gluster
11:14 partner hmm well i guess its fine now, traffic jumped from few bps to 150 Mbps
11:14 partner and been steady since so its really sucking in files and also serving about same amount, disk utilization up from ~0 to ~50%
11:25 rjoseph joined #gluster
11:25 partner joined #gluster
11:26 andreask joined #gluster
11:36 skered- joined #gluster
11:36 RameshN joined #gluster
11:37 anands joined #gluster
11:37 pithagorians joined #gluster
11:38 pithagorians hello all. why 3.4.1 requests multiarch-support on debian 6 /
11:38 pithagorians ?
11:39 pithagorians 64 bit
11:42 _polto_ joined #gluster
11:42 _polto_ joined #gluster
11:43 RameshN joined #gluster
11:45 jtux joined #gluster
11:48 davidbierce joined #gluster
11:51 partner hmm where did you get packages for squeeze?
11:53 pithagorians http://download.gluster.org/pub/gluster/glust​erfs/LATEST/Debian/apt/pool/main/g/glusterfs/
11:53 glusterbot Title: Index of /pub/gluster/glusterfs/LATEST/D​ebian/apt/pool/main/g/glusterfs (at download.gluster.org)
11:53 pithagorians here
11:53 partner that's for wheezy
11:54 pithagorians probably
11:54 pithagorians here its nor clear
11:54 pithagorians i need 3.4 for squeeze
11:54 pithagorians :)
11:54 partner there is no such out there
11:54 pithagorians so i need wheezy to be able to install 3.4 ?
11:54 pithagorians :D
11:55 partner i asked and got all the required files available to that location to package it for squeeze
11:55 partner and built it for myself for squeeze
11:55 pithagorians hmm
11:56 partner requires some minor touches to the dependencies and off you go
11:57 pithagorians hm
11:57 pithagorians seams like i will go with 3.3
11:58 pithagorians i wanted to have the 3.4 because there were fixed some bugs
12:00 partner yeah. i still need to build 3.3.2 for squeeze clients as that was available for wheezy only aswell thought i might go to 3.4 already, testing in production...
12:03 partner i know someone is thinking in the background why the heck we don't upgrade but its just not that easy to go and upgrade when the service running on top of the box does not support it..
12:04 partner doing it, takes time, 500+ custom services here and there so its not one days effort exactly
12:16 morse joined #gluster
12:17 psyl0n joined #gluster
12:18 getup- joined #gluster
12:24 dusmant joined #gluster
12:25 rjoseph joined #gluster
12:33 dusmant joined #gluster
12:46 _pol joined #gluster
12:48 dusmantkp_ joined #gluster
13:11 dusmantkp_ joined #gluster
13:12 abyss^ a long time is waiting for approval message to gluster-users@gluster.org? I send message 1h ago and I wonder about that;)
13:12 _polto_ joined #gluster
13:26 edward2 joined #gluster
13:31 dusmantkp_ joined #gluster
13:35 satheesh joined #gluster
13:37 davidbierce joined #gluster
13:39 B21956 joined #gluster
13:41 kkeithley abyss^: wellllll. It's late, very late, Friday night in Bangalore. And it's too early in the morning in the U.S. for most people. I'd give it more time.
13:45 bennyturns joined #gluster
13:47 mkzero joined #gluster
13:48 anands joined #gluster
13:50 abyss^ kkeithley: ah, you're right. Thx:)
13:52 muhh joined #gluster
13:59 badone joined #gluster
14:09 mistich anyone seen where process hang with gluster and dmesg spits out this?
14:09 mistich INFO: task python:16844 blocked for more than 120 seconds.
14:09 mistich Not tainted 2.6.32-431.el6.x86_64 #1
14:09 mistich "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
14:09 mistich python        D 0000000000000000     0 16844      1 0x00000080
14:09 mistich ffff88065d31be38 0000000000000082 ffff88065d31be88 ffff88078b18c3a8
14:09 mistich ffff88078b18c500 0000000300000001 00000000000000d0 ffff88080f7b0840
14:09 mistich ffff880812c93098 ffff88065d31bfd8 000000000000fbc8 ffff880812c93098
14:09 mistich Call Trace:
14:09 mistich [<ffffffff8109b5ce>] ? prepare_to_wait+0x4e/0x80
14:09 mistich [<ffffffffa017b085>] fuse_request_send+0xe5/0x290 [fuse]
14:09 mistich [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
14:09 mistich [<ffffffffa0180ff6>] fuse_flush+0x106/0x140 [fuse]
14:10 Staples84 joined #gluster
14:15 foster mistich: that probably means there is a request in flight to gluster that hasn't been completed
14:15 foster close(), in this case
14:16 mistich any suggestion on fixing?
14:18 foster not really enough information, we'd need to figure out what's happening with that request in gluster
14:19 mistich what info would I need and where would I send it to?
14:19 foster is this something you can reproduce?
14:19 mistich oh yeah
14:20 mistich sick of restarting the process
14:20 foster heh, then please file a bug
14:20 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
14:22 mattapp__ joined #gluster
14:23 plarsen joined #gluster
14:23 mistich you think it is a bug
14:23 japuzzo joined #gluster
14:24 mistich I have found some old post like this issue in 3.2.0. http://gluster.org/pipermail/glu​ster-users/2011-June/007984.html
14:24 glusterbot Title: [Gluster-users] Apache hung tasks still occur with glusterfs 3.2.1 (at gluster.org)
14:25 foster well I doubt it's a feature ;)
14:25 rjoseph joined #gluster
14:26 mistich :)
14:27 FilipeCifali joined #gluster
14:28 FilipeCifali do like to thank everyone who helped me these past days, the setup is now properly working in 3.3.2 and will go to production in 30 days after beta period :)
14:30 FilipeCifali joined #gluster
14:35 dbruhn joined #gluster
14:38 bsaggy joined #gluster
14:47 mattappe_ joined #gluster
14:49 zerick joined #gluster
14:49 hagarth joined #gluster
14:50 matta____ joined #gluster
14:52 davinder joined #gluster
14:53 failshell joined #gluster
14:57 _BryanHm_ joined #gluster
15:13 spechal_ left #gluster
15:16 wushudoin joined #gluster
15:17 bet_ joined #gluster
15:20 ira joined #gluster
15:20 ira joined #gluster
15:22 kaptk2 joined #gluster
15:22 abyss^ Let's assume my first gluster (that where clients are connected in fstab) crashed totaly, no data etc, but second gluster still working... Can I use replace-brick command to make new gluster first?
15:24 lpabon joined #gluster
15:30 TrDS left #gluster
15:32 Technicool joined #gluster
15:34 rjoseph joined #gluster
15:34 Staples84 joined #gluster
15:35 ndk joined #gluster
15:36 lkoranda_ joined #gluster
15:56 harish joined #gluster
16:04 neofob joined #gluster
16:05 jag3773 joined #gluster
16:07 failshell joined #gluster
16:11 johnbot11 joined #gluster
16:11 jbrooks Technicool: I have couple Q's about the gluster quickstart -- we should include the firewall ports, right? And, wouldn't it be quicker to not set up an xfs partition, and just do it in a file, or is that too messed up even for a Hello World scenario?
16:11 jbrooks I mean, just do it in a folder, versus an xfs part
16:12 Technicool re: XFS, i think its important to handhold folks since they tend to get hung up on being able to conceptualize that doing things in / is not a great idea real world
16:12 Technicool firewall ports need to be updated as of 3.4
16:12 lava joined #gluster
16:12 jbrooks I think the quickstart hasn't mentioned firewall, though
16:13 Technicool i thought it did?  or are you talking about the really really quickstart?  ;)
16:13 Technicool if not, yes, it could use that
16:15 jbrooks Maybe we could include instructions for doing xfs in a loopback device, optionally
16:16 andreask joined #gluster
16:16 Technicool jbrooks, yes that is also a good solution
16:16 gmcwhistler joined #gluster
16:29 LoudNoises joined #gluster
16:34 shyam joined #gluster
16:42 theron joined #gluster
16:47 rjoseph joined #gluster
16:53 jbd1 joined #gluster
16:53 bgpepi joined #gluster
16:54 JonathanD joined #gluster
16:55 yosafbridge joined #gluster
16:56 JoeJulian joined #gluster
17:01 vimal joined #gluster
17:04 piffio joined #gluster
17:14 mkzero joined #gluster
17:19 jbd1 joined #gluster
17:33 davinder joined #gluster
17:34 Mo__ joined #gluster
17:34 SpeeR joined #gluster
17:50 _pol joined #gluster
17:53 anands joined #gluster
17:58 anands left #gluster
18:04 rotbeard joined #gluster
18:05 zaitcev joined #gluster
18:09 SpeeR can anyone tell me how gluster chooses which brick to write files to? I have a 4 brick setup, and 1 brick is at 100% usage while the others are at 20-50% usage
18:10 semiosis assuming you mean how does distribute decide, it does so by hashing the file name
18:15 johnmwilliams joined #gluster
18:16 SpeeR so in a vmware environment, does it try to keep all VM files on the same brick in the directory with all of the other files?
18:17 semiosis not really
18:17 rwheeler joined #gluster
18:17 sarkis joined #gluster
18:17 SpeeR ok thanks semiosis, I'll rebalance, and keep an eye on it
18:24 JoeJulian SpeeR: You can see how the algorithm works at http://joejulian.name/blog​/dht-misses-are-expensive/
18:24 glusterbot Title: DHT misses are expensive (at joejulian.name)
18:24 kkeithley vmware or otherwise, short, similar file names will be very likely to hash to the same brick.
18:24 SpeeR hah I was just on your blog looking at your .glusterfs dir info
18:26 MacWinner I have a 4-node cluster with replica 2.. the 4th node's seocnd hard drive  died and I swapped it out.  it's now back online and reformed with xfs.. will it automatically heal? or do I need to do something?
18:26 JoeJulian I thought the whole idea of hashing was that similar names should still give a statistically distributed result.
18:26 JoeJulian MacWinner: You'll want to "gluster volume heal $vol full"
18:27 SpeeR yeh, I could see similar file names/sizes could be sent to the same brick depending on how it's hashed
18:30 kkeithley sure, but then statistics get involved.
18:30 kkeithley IIRC jdarcy wrote a blog about it.
18:37 kkeithley But I do know that we occasionally see pathologically unbalanced bricks and the full bricks usually have a lot of files with short, similar file names.
18:37 MacWinner JoeJulian, hmm.. tried it, and got error message: "Commit failed on chi3-internal. Please check the log file for more details."  chi4 is the the 4th node that i'm trying to heal..  i checked the logs, but didn't see anything specific
18:38 MacWinner i can pastebin the logs
18:39 MacWinner i do see this error though: "[2013-12-06 18:39:14.007547] W [socket.c:514:__socket_rwv] 0-storage-client-3: readv failed (No data available)"  not sure why it's showing up though
18:41 MacWinner "gluster volume heal storage info" indicates "Status: Brick is Not connected" on node4.. so maybe need to do something to reconnect the brick prior to calling heal?
18:45 MrNaviPacho joined #gluster
18:45 JoeJulian is glusterd started on 4?
18:47 MacWinner yep, i tried restarting it too
18:47 JoeJulian check peer status on the other server(s)
18:48 MacWinner they all say chi4 is connected
18:48 MacWinner "State: Peer in Cluster (Connected)"
18:48 JoeJulian I think I know why...
18:49 JoeJulian There's probably a brick log on 4 for the "missing" brick. Check that.
18:50 MacWinner http://pastebin.com/4mL6cdp6
18:50 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
18:51 MacWinner that's the brick.log and volume info on chi4
18:51 MacWinner looks like this is the offending line: "[posix.c:4288:init] 0-storage-posix: Extended attribute trusted.glusterfs.volume-id is absent"
18:52 MacWinner oh..i see your blog post on it :)
18:52 JoeJulian That's what I was suspecting.
18:52 JoeJulian Oh, I posted on that? ;)
18:53 MacWinner http://joejulian.name/blog/repl​acing-a-brick-on-glusterfs-340/
18:53 glusterbot Title: Replacing a brick on GlusterFS 3.4.0 (at joejulian.name)
18:53 JoeJulian heh
18:53 JoeJulian I forgot I did that.
18:57 MacWinner JoeJulian, thanks man.. that fixed it!
19:01 failshel_ joined #gluster
19:03 kkeithley Gluster Test Day: YUM repos with 3.5.0qa3 RPMs for EPEL-6 and Fedora 18, 19, and 20  are available at http://download.gluster.org/pub/glus​ter/glusterfs/qa-releases/3.5.0qa3/
19:03 glusterbot Title: Index of /pub/gluster/glusterfs/qa-releases/3.5.0qa3 (at download.gluster.org)
19:05 failshe__ joined #gluster
19:06 kkeithley I wonder why glusterbot didn't do it's thing in #gluster-devel
19:08 _pol__ joined #gluster
19:12 andreask joined #gluster
19:18 semiosis kkeithley: most supybot features are enabled & configured per channel
19:20 semiosis kkeithley: have you seen this?
19:20 semiosis https://bugzilla.redhat.co​m/show_bug.cgi?id=1038391
19:20 glusterbot Bug 1038391: unspecified, unspecified, ---, amarts, NEW , gf-error-codes.h is missing from source tarball
19:20 semiosis 3.5.0qa3 did not require running autogen.sh
19:20 semiosis actually never mind
19:20 semiosis it might
19:20 semiosis but still curious if you have any thoughts on that bug
19:22 Gilbs1 joined #gluster
19:22 kkeithley Saw it. Sorry to say I haven't looked into it. I do all my builds with source from git or rpmbuilds, not from the release tarball.
19:26 kkeithley I'll try to get to it soon
19:27 kkeithley If someone else doesn't get to it first
19:27 sgowda joined #gluster
19:27 theron joined #gluster
19:29 Gilbs1 Anyone have issues stopping geo-replication?  Everything is syncing, but I'd like to stop it and I keep getting: geo-replication command failed   (3.4/centos 6)
19:36 glusterbot New news from resolvedglusterbugs: [Bug 950083] Merge in the Fedora spec changes to build one single unified spec <https://bugzilla.redhat.com/show_bug.cgi?id=950083>
19:44 ira joined #gluster
19:45 davidbierce Does the NUFA translator work with Distribute Replicate?  Or more specifically, I know writes would still be slow, but it would be handy if reads didn't have to go over the network for situations where usage of the data would be local.  Like a VM host running VMs on gluster.
19:52 mattapp__ joined #gluster
19:58 _pol joined #gluster
19:58 NigeyS joined #gluster
19:58 NigeyS evening :)
20:04 achuz joined #gluster
20:05 gdavis331 left #gluster
20:08 failshell joined #gluster
20:11 mattapp__ joined #gluster
20:28 pravka joined #gluster
20:32 XpineX joined #gluster
20:32 sprachgenerator joined #gluster
20:36 SpeeR ok, I'm back. I've done a rebalance, however I haven't freed up any space on the full brick http://ur1.ca/g5k97
20:36 glusterbot Title: #59686 Fedora Project Pastebin (at ur1.ca)
20:36 SpeeR should this only be a link now?
20:37 SpeeR or could it still be in flight? The balance is showing completed
20:37 elyograg SpeeR: when you are dealing with really large files like VM images, an imbalance in space usage is likely.
20:38 rjoseph joined #gluster
20:39 SpeeR elyograg, I can see that, however this particular file, is eating up 8TB on 2 bricks with a tar that appears to be identical
20:40 SpeeR I can see that happening that is, there was no snark there even though it might sound like it
20:40 SpeeR heh
20:54 tqrst heh, just realized someone had a check out of the entire chromium source tree on our gluster volume.
20:54 tqrst totally sensible place to put 57000+ folders filled with tiny files
21:07 daMaestro joined #gluster
21:09 hchiramm_ joined #gluster
21:09 NigeyS would anyone know if the gluster client still has a secondary group limit of 15  ?
21:09 Gilbs1 left #gluster
21:14 semiosis @ppa
21:14 glusterbot semiosis: The official glusterfs packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.4 stable: http://goo.gl/u33hy
21:14 semiosis @forget ppa
21:14 glusterbot semiosis: The operation succeeded.
21:15 semiosis @learn ppa as The official glusterfs packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.4 stable: http://goo.gl/u33hy -- 3.5 QA: http://goo.gl/Odj95k
21:15 glusterbot semiosis: The operation succeeded.
21:15 semiosis @ppa
21:15 glusterbot semiosis: The official glusterfs packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.4 stable: http://goo.gl/u33hy -- 3.5 QA: http://goo.gl/Odj95k
21:15 neofob left #gluster
21:21 semiosis davidbierce: i dont think NUFA is used in modern versions of glusterfs
21:24 johnbot11 joined #gluster
21:25 davidbierce semiosis: that is kind of what I thought, but was mainly looking for confirmation.  Was just thinking for some of our traffic, it would be handy to have reads not traverse the network if they don't have to.  But there would have to be a way to influence balancing as well if files moved around :(
21:26 rjoseph joined #gluster
21:31 _BryanHm_ joined #gluster
21:41 rotbeard joined #gluster
22:04 zerick joined #gluster
22:04 johnbot11 joined #gluster
22:10 sprachgenerator joined #gluster
22:10 theron joined #gluster
22:36 eclectic joined #gluster
22:37 Oneiroi joined #gluster
22:42 hchiramm_ joined #gluster
22:49 davidbierce joined #gluster
22:59 jag3773 joined #gluster
23:05 psyl0n joined #gluster
23:09 sticky_afk joined #gluster
23:09 stickyboy joined #gluster
23:23 hchiramm_ joined #gluster
23:25 johnbot11 joined #gluster
23:31 MrNaviPacho joined #gluster
23:33 _pol joined #gluster
23:34 _pol_ joined #gluster
23:44 bala1 joined #gluster
23:48 fyxim joined #gluster
23:59 mattapp__ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary