Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-03-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 kam270_ joined #gluster
00:08 chirino joined #gluster
00:12 kam270_ joined #gluster
00:16 robo joined #gluster
00:17 RicardoSSP joined #gluster
00:17 RicardoSSP joined #gluster
00:19 m0zes joined #gluster
00:25 primechuck joined #gluster
00:29 bala joined #gluster
00:29 m0zes joined #gluster
00:34 m0zes joined #gluster
00:39 m0zes_ joined #gluster
00:44 m0zes joined #gluster
00:49 m0zes_ joined #gluster
00:49 neurodrone__ joined #gluster
00:54 m0zes joined #gluster
01:06 yinyin joined #gluster
01:08 primechu_ joined #gluster
01:10 chirino joined #gluster
01:25 neurodrone__ joined #gluster
01:27 tokik joined #gluster
01:30 Ark joined #gluster
01:35 harish joined #gluster
01:39 calum_ joined #gluster
01:39 chirino joined #gluster
01:54 ultrabizweb joined #gluster
01:58 Jankies joined #gluster
01:58 cjanbanan joined #gluster
02:03 Jankies left #gluster
02:03 Jankies joined #gluster
02:23 coredum__ joined #gluster
02:25 nueces joined #gluster
02:30 Jakey joined #gluster
02:51 shubhendu joined #gluster
02:56 nueces_ joined #gluster
03:02 hagarth joined #gluster
03:39 wrale_ joined #gluster
03:43 DV joined #gluster
03:54 itisravi joined #gluster
04:05 ndarshan joined #gluster
04:11 yinyin joined #gluster
04:12 kanagaraj joined #gluster
04:17 Psi-Jack joined #gluster
04:20 Psi-Jack So, I just had a horrible case of splitbrain.
04:21 Psi-Jack qcow2 disk image was split-brained, and I tried to delete the failing copy off the brick and self-heal, nothing, wouldn't self heal. Only healing I could do was rsync the file to a new location off the brick, delete the hard links, and re-rsync back onto the gluster mount again
04:23 ravindran joined #gluster
04:27 prasanthp joined #gluster
04:43 chirino joined #gluster
04:44 dusmant joined #gluster
04:47 kshlm joined #gluster
04:49 bala joined #gluster
05:03 deepakcs joined #gluster
05:07 lalatenduM joined #gluster
05:07 satheesh joined #gluster
05:10 kdhananjay joined #gluster
05:12 RameshN joined #gluster
05:13 chirino joined #gluster
05:14 slayer192 joined #gluster
05:19 yinyin joined #gluster
05:23 discrete_ joined #gluster
05:24 nshaikh joined #gluster
05:32 sks joined #gluster
05:38 satheesh joined #gluster
05:43 aravindavk joined #gluster
05:54 dusmant joined #gluster
05:58 psharma joined #gluster
06:01 shylesh joined #gluster
06:07 efries joined #gluster
06:08 gdubreui joined #gluster
06:11 benjamin_____ joined #gluster
06:15 chirino joined #gluster
06:15 Philambdo joined #gluster
06:18 vimal joined #gluster
06:38 FarbrorLeon joined #gluster
06:46 dusmant joined #gluster
07:01 ngoswami joined #gluster
07:05 lalatenduM joined #gluster
07:13 calum_ joined #gluster
07:16 satheesh1 joined #gluster
07:17 rahulcs joined #gluster
07:21 ricky-ti1 joined #gluster
07:34 XATRIX joined #gluster
07:38 jtux joined #gluster
07:42 andreask joined #gluster
07:49 slayer192 joined #gluster
07:53 ekuric joined #gluster
07:56 DV joined #gluster
07:56 eseyman joined #gluster
07:59 ekuric left #gluster
07:59 ekuric joined #gluster
08:00 ctria joined #gluster
08:05 ade_b joined #gluster
08:13 trendzettter joined #gluster
08:15 cjanbanan joined #gluster
08:25 trendzettter @semiosis: I am trying to use your gluster enabled qemu build from the ubuntu-qemu-glusterfs ppa. I installed qemu, qemu-kvm, glusterfs-client and glusterfs-server from the ppa.
08:26 ProT-0-TypE joined #gluster
08:27 trendzettter I am using libvirt 1.2.2 from trusty
08:28 trendzettter I am now getting Unknown protocol 'gluster://localhost
08:29 trendzettter I am wondering if you could give some instructions on how to get the gluster-enabled qemu functional.
08:35 fsimonce joined #gluster
08:47 liquidat joined #gluster
08:47 hagarth joined #gluster
08:48 hagarth trendzettter: this might help - http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems
08:48 glusterbot Title: Building QEMU with gfapi for Debian based systems - GlusterDocumentation (at www.gluster.org)
08:53 ndevos trendzettter: there are two parts to that, the first would be to check if libvirt knows about Gluster - see http://blog.nixpanic.net/2013/11/initial-work-on-gluster-integration.html for some notes on that
08:53 glusterbot Title: Nixpanic's Blog: Initial work on Gluster integration with CloudStack (at blog.nixpanic.net)
08:54 ndevos trendzettter: that also includes a way to create a disk through the libvirt shell (virsh) and shows the qemu(-kvm) command to use the disk
08:55 trendzettter @hagarth: Thanks for your response, however. semiosis seems to have ready made packages so I don't have to build myself (https://launchpad.net/~semiosis/+archive/ubuntu-qemu-glusterfs)
08:55 lalatenduM hagarth, ping
08:55 hagarth lalatenduM: pong
08:55 lalatenduM hagarth, I created this page but not able to edit it somehow http://www.gluster.org/community/documentation/index.php/Bug_report_life_cycle
08:55 glusterbot Title: Bug report life cycle - GlusterDocumentation (at www.gluster.org)
08:56 lalatenduM hagarth, not sure what went wrong
08:56 hagarth lalatenduM: are you logged out per chance?
08:56 al joined #gluster
08:56 lalatenduM hagarth, nope, I am logged in , can edit other pages
08:57 lalatenduM hagarth, I think I missed something in the media wiki format
08:57 hagarth lalatenduM: I am able to edit that page
08:57 lalatenduM hagarth, hmm may be you are Administrator or something , I need to attach a image to the page too
08:58 jbustos joined #gluster
08:59 ndevos lalatenduM: you (and I) are probably not in the uploadaccess group, mayby hagarth or jclift can add us?
08:59 ndevos *maybe
08:59 * ndevos would like to be able to upload his presentations in the wiki
08:59 lalatenduM ndevos, agree, I think so
09:00 dusmant joined #gluster
09:00 lalatenduM ndevos, I am writing this in the wiki http://www.gluster.org/community/documentation/index.php/Bug_triage
09:00 glusterbot Title: Bug triage - GlusterDocumentation (at www.gluster.org)
09:00 ndevos lalatenduM: also, I was wondering if you have thought about cloning (or not) bugs from the master branch back to a release? we now often have one bug for multiple releases and that makes tracking difficult
09:00 hagarth ndevos, lalatenduM: done  - both of you have been added to uploadaccess
09:01 ndevos hagarth: thanks!
09:01 lalatenduM hagarth, Thanks!
09:01 raghu joined #gluster
09:01 kanagaraj joined #gluster
09:02 lalatenduM ndevos, , yup, something to think about, we should put that in the triage page and can take a feedback from others
09:03 ndevos lalatenduM: can you add that to some TODO/needinfo/thinkabout list?
09:03 lalatenduM ndevos, sure, will do
09:03 ndevos thanks!
09:04 sks joined #gluster
09:05 lalatenduM hagarth, I uploaded the file, but still not able to edit the page :( ndevos ^^ http://www.gluster.org/community/documentation/index.php/Bug_report_life_cycle
09:05 glusterbot Title: Bug report life cycle - GlusterDocumentation (at www.gluster.org)
09:05 keytab joined #gluster
09:06 ndevos lalatenduM: I'm able to edit it...
09:07 lalatenduM ndevos, I am not not able to see the edit link itself
09:07 lalatenduM I mean the edit hyperlink
09:07 ndevos lalatenduM: you have a 'show source' tab or something?
09:08 ndevos lalatenduM: http://www.gluster.org/community/documentation/index.php?title=Bug_report_life_cycle&action=edit is the link for editing
09:08 glusterbot Title: View source for Bug report life cycle - GlusterDocumentation (at www.gluster.org)
09:08 lalatenduM ndevos, checking
09:08 ndevos right, glusterbot isnt logged in on the wiki, he/she should only be able to view the source
09:09 lalatenduM hagarth, ndevos, yup working
09:09 hagarth lalatenduM: cool
09:09 lalatenduM ndevos, hagarth thanks :)
09:12 RameshN joined #gluster
09:13 ndarshan joined #gluster
09:21 kiwnix joined #gluster
09:25 primechuck joined #gluster
09:26 hybrid512 joined #gluster
09:26 hybrid512 joined #gluster
09:28 dusmant joined #gluster
09:28 kanagaraj joined #gluster
09:32 badone joined #gluster
09:34 hagarth joined #gluster
09:36 ctria joined #gluster
09:41 rahulcs_ joined #gluster
09:41 ThatGraemeGuy_ joined #gluster
09:42 layer3switch joined #gluster
09:43 satheesh1 left #gluster
09:52 glusterbot New news from resolvedglusterbugs: [Bug 1073071] Glusterfs 3.4.2 data replication doesn't work for cinder backend in RDO Havana on Fedora 20 two node cluster <https://bugzilla.redhat.com/show_bug.cgi?id=1073071>
09:57 monotek @semiosis
09:57 glusterbot monotek: I do not know about 'semiosis', but I do know about these similar topics: 'semiosis tutorial'
09:57 monotek are you there?
10:10 keytab joined #gluster
10:15 badone joined #gluster
10:17 ninkotech joined #gluster
10:17 ninkotech_ joined #gluster
10:29 ndevos monotek: there is no use in putting the @ in front it nicknames, the symbol is used for provoking a response from glusterbot
10:30 monotek sorry. didnt know... thanks for info :-)
10:31 ndevos no problem, monotek
10:33 gdubreui joined #gluster
10:33 diegows joined #gluster
10:39 ProT-0-TypE joined #gluster
10:43 primechuck joined #gluster
10:44 primechu_ joined #gluster
10:49 primechuck joined #gluster
10:50 ProT-0-TypE joined #gluster
10:56 ctria joined #gluster
11:04 tdasilva left #gluster
11:14 kdhananjay1 joined #gluster
11:19 kanagaraj joined #gluster
11:20 dusmant joined #gluster
11:22 bala joined #gluster
11:24 ndarshan joined #gluster
11:25 XATRIX joined #gluster
11:25 XATRIX Hi, how can i what if i have my shared storage and a mount point on 2 nodes
11:25 XATRIX Oh, sorry
11:25 XATRIX how can i enable direct-io=enable ?
11:26 XATRIX and will it help me a bit ? with my setup
11:26 XATRIX 1node[raid1(2hdd)] + 2node[raid1(2hdd)]
11:48 hagarth joined #gluster
11:51 chirino joined #gluster
12:02 doekia @XATRIX: I guess this is the performance.strict-o-direct parameter (gluster volume set xxx performance.strict-o-direct on)
12:03 al joined #gluster
12:04 XATRIX doekia: any idea where can i find the whole list of these parameters ?
12:06 doekia the whole is not really avail I use a mix of gluster volume set help & http://www.gluster.org/community/documentation/index.php/Documenting_the_undocumented
12:06 glusterbot Title: Documenting the undocumented - GlusterDocumentation (at www.gluster.org)
12:06 XATRIX Also, is there any sense to enable direct-io on my setup ?
12:06 doekia And at last I browse the source
12:06 XATRIX I mean i have 2 nodes in a cluster, and 2x hdds in a raid1 on each node
12:06 XATRIX i also mount the fs on both nodes
12:10 itisravi joined #gluster
12:23 jiffe98 so I'm in the process of upgrading the rest of my nodes
12:23 jiffe98 I have all but one upgraded but the last one I upgraded isn't starting saying Peer gs2 does not support required op-version
12:24 Philambdo joined #gluster
12:24 al joined #gluster
12:28 dan joined #gluster
12:29 Ark joined #gluster
12:30 neurodrone joined #gluster
12:30 neurodrone joined #gluster
12:30 harish joined #gluster
12:37 saurabh joined #gluster
12:38 jag3773 joined #gluster
12:39 rfortier1 joined #gluster
12:39 RameshN joined #gluster
12:41 tokik joined #gluster
12:45 edward1 joined #gluster
12:50 sroy joined #gluster
12:51 DV joined #gluster
12:54 jiffe98 alright, got 3 of the 4 nodes upgraded, last one I'll do tonight
13:01 robo joined #gluster
13:02 Pavid7 joined #gluster
13:05 robo joined #gluster
13:17 kanagaraj joined #gluster
13:18 ndarshan joined #gluster
13:19 tdasilva joined #gluster
13:20 robo joined #gluster
13:22 RameshN joined #gluster
13:22 chirino joined #gluster
13:23 bala joined #gluster
13:27 jmarley joined #gluster
13:27 jmarley joined #gluster
13:29 ndk joined #gluster
13:29 jtux joined #gluster
13:31 dbruhn joined #gluster
13:35 dewey joined #gluster
13:38 theron joined #gluster
13:38 dusmant joined #gluster
13:40 jiffe98 hmm, noticed the last peer I upgraded isn't connecting to the other two nodes I upgraded http://nsab.us/public/gluster
13:44 DV joined #gluster
13:47 japuzzo joined #gluster
13:48 failshell joined #gluster
13:54 bennyturns joined #gluster
14:01 seapasulli joined #gluster
14:02 diegows joined #gluster
14:16 nshaikh joined #gluster
14:19 chirino_m joined #gluster
14:25 rpowell joined #gluster
14:27 davinder joined #gluster
14:29 Pavid7 joined #gluster
14:30 skryzhny joined #gluster
14:33 robo joined #gluster
14:34 rakkaus_ joined #gluster
14:40 jobewan joined #gluster
14:46 discretestates joined #gluster
14:47 lmickh joined #gluster
14:50 coredump_ joined #gluster
14:50 skryzhny Hi All! I have problems updating gluster 3.1.2 (Fedora15) to gluster 3.4.2 (Centos6 glusterfs-epel repo)
14:50 skryzhny after updating RPMs I get Staging of operation 'Volume Start' failed on localhost : Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /export/brick0. R
14:50 skryzhny eason : No data available
14:52 skryzhny I'm read http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3/
14:53 skryzhny did anyone successfully updated directly from 3.1 to 3.4 ?
14:54 shylesh joined #gluster
14:59 kdhananjay joined #gluster
14:59 skryzhny I have *.vol.rpmsave old configs and updated *.vol files, so I think rpms installation was successful
14:59 sahina joined #gluster
15:12 sks joined #gluster
15:17 georgeh|workstat joined #gluster
15:18 daMaestro joined #gluster
15:26 sprachgenerator joined #gluster
15:27 Pavid7 joined #gluster
15:37 lpabon joined #gluster
15:37 mattappe_ joined #gluster
15:40 rshade98 did you mess with the extended file attributes?
15:41 zaitcev joined #gluster
15:42 skryzhny I think no, this bricks worked OK with glusterfs 3.1.2
15:43 rshade98 when I restored a volume between 3.1 and 3.4 I had to do this: http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/
15:43 glusterbot Title: GlusterFS: {path} or a prefix of it is already part of a volume (at joejulian.name)
15:43 rshade98 not sure if its the same thing though
15:43 mattappe_ joined #gluster
15:44 benjamin_____ joined #gluster
15:45 skryzhny I tried downgrade glusterfs to 3.3.1, and it looks like it worked, I only need to "setfattr -x trusted.gfid /export/brick1"
15:50 chirino joined #gluster
15:54 RayS joined #gluster
15:59 theron joined #gluster
16:00 theron_ joined #gluster
16:00 jmarley joined #gluster
16:00 jmarley joined #gluster
16:02 hagarth joined #gluster
16:04 social joined #gluster
16:16 ade_b joined #gluster
16:17 ade_b is "Another transaction could be in progress. Please try again after sometime." something bad, or should I just wait
16:17 ade_b (when typing gluster volume status)
16:18 lalatenduM ade_b, are you running two cli commands simultaneously from diff nodes?
16:18 mattappe_ joined #gluster
16:18 DV joined #gluster
16:19 ade_b well yes, but it completed before I went to the 2nd node
16:20 jag3773 joined #gluster
16:20 ade_b lalatenduM, and there is about a 2gig difference in the brick sizes (16gig and 18gig) :-|
16:20 seapasulli joined #gluster
16:21 ade_b lalatenduM, gluster peer status seems to suggest they are both connected
16:21 lalatenduM ade_b, the brick size diff is not an issue
16:22 ade_b lalatenduM, phew
16:22 lalatenduM ade_b, does the command runs fine after sometime , I mean after you get the error
16:22 ade_b there are setup to replicate , so I exected them to stay identical
16:23 ade_b lalatenduM, well Im retrying from time to time but I have just noticed I have a zombie process too
16:23 lalatenduM ade_b, gluster allows one command at a time
16:23 lalatenduM for whole cluster
16:23 rwheeler joined #gluster
16:23 vpagan joined #gluster
16:23 ade_b ok, so I should wait
16:24 Mo_ joined #gluster
16:24 ade_b lalatenduM, commands on the other node seem to work just fine
16:24 vpagan I have a rebalance question if anyone is available
16:26 ade_b lalatenduM, ah, now the brick sizes are back the same
16:26 lalatenduM ade_b, I dont understand :( , how brick size is changing for u
16:27 ade_b lalatenduM, I was thinking it was a good sign :-/
16:28 lalatenduM ade_b, bricks should be separate partitions , are talking abt availabe size, used size or total size?
16:29 lalatenduM vpagan, u should ask the question , if anybody knows abt it, u will get the ans
16:29 ade_b I do a du -sh /data/brick1 on one node and then compare this to the 2nd node
16:30 vpagan sure thing. I'm doing a rebalance on two distributed bricks and none of the data is actually being rebalanced, just the structure
16:31 vpagan both are 9T bricks, one with 70GB used and the other with 600MB used. Rebalance force doesn't solve the issue either
16:31 lalatenduM ade_b, du shows used size, so thats ans the question , used size can be diff for bricks
16:32 ade_b ok, so I shouldnt worry if there is a difference - even if they are replicas?
16:33 kshlm vpagan, rebalance without adding a new-brick wouldn't do anything, as the data is already where it is supposed to be.
16:33 lalatenduM ade_b, ideallu replicas have same used size, but it can differ if one brick of replica pair was down i.e. it needs to be healed
16:33 vpagan sorry, I should've mentioned the 600mb brick is a newly added brick
16:33 lalatenduM ade_b, one self-heal is complete , it should be of same size
16:33 lalatenduM s/one/once/
16:34 glusterbot What lalatenduM meant to say was: ade_b, once self-heal is complete , it should be of same size
16:34 vpagan it was a brick that I removed, rebuilt to increase drive sizes, and added to the volume a gain
16:35 ade_b lalatenduM, glusterbot thanks
16:37 kshlm is any new data you are adding, being written to the new brick?
16:37 vpagan Let me test that
16:38 vpagan yes, it is going to the new brick (600mb one)
16:41 kshlm in that case (AFAIK) the 70GB data you have in brick1 is meant to be there.
16:41 vpagan what do you mean by meant to be there?
16:41 kshlm Gluster uses hashing bsaed on filenames to distribute data. This could sometime lead to uneven distribution.
16:42 Joe630 not if it is hashed
16:42 Joe630 and using a proper hashing function
16:43 vpagan sure, but why would this affect a rebalance
16:44 ndevos Joe630: the hash is based on the filename, not including the whole directory structure - so a file called README in one dir, results in the same hash as README in an other dir
16:45 ndevos (well, there is a twist in the hashing function, so it will not place all README files on the same brick)
16:45 kshlm rebalance makes sure that the files are in the place where they are hashed to. if the file is not in the hashed location rebalance will move it there.
16:45 Joe630 are you sure it is just the filename and not the whole path?
16:46 kshlm Joe630, yup just the filename.
16:46 Joe630 that is broken.
16:46 ndevos Joe630: yes, but I'm not sure how the twist works, I think it has to do with the directory dept
16:46 Joe630 if this were my code, i'd expect a bug report.
16:46 kshlm not so fast.
16:46 ndevos Joe630: doing a lookup for all the parent directories would make it slower...
16:46 Joe630 the exception being if the file is the same and it de-dupes it.  Which it doesn't.
16:47 kshlm each directory has a different layout for distributing the files in it.
16:47 kshlm so even if two directories contain a file with the same name, they will end up in different locations because of the different layouts.
16:47 Joe630 well, how many files have the same name in a real environment
16:47 Joe630 a.out README INSTALL
16:48 Joe630 IU still think thats a bad idea™
16:48 vpagan kshlm, you brought something to my attention
16:50 kshlm Joe630, you can read jdarcy's writeup on the distribution algorithm http://webcache.googleusercontent.com/search?q=cache:kEeSBSbT0qkJ:hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/+&amp;cd=1&amp;hl=en&amp;ct=clnk&amp;gl=in
16:50 vpagan as I remove-brick one of the bricks, it never fully moved all of the data. So I rsync'd the rest over. I'm going to assume that the hashes are kept in that .glusterfs folder? Which is why the 70GB server thinks it owns those files and isn't rebalancing them
16:51 kshlm the hashes aren't stored anywhere. they are always calculated when needed based on the filename.
16:51 chirino_m joined #gluster
16:52 vpagan then I'm still confused as to why those files never got rebalanced afterwards
16:52 kshlm did you rsync the new remaining files directly into the brick folder?
16:52 kshlm s/new //
16:52 glusterbot What kshlm meant to say was: did you rsync the remaining files directly into the brick folder?
16:53 Joe630 vpagan: i was told to never.ever.ever. write directly to the FS
16:53 Joe630 and in my experience, they never are replicated when I do that.
16:54 Joe630 the reasons given were that gluster uses extended attributes in xfs
16:54 kshlm Joe630, correct.
16:54 vpagan let me clarify my setup
16:54 vpagan I have two debian servers running gluster and zfs
16:54 Joe630 so if you rsynced them, it wouldn't have coppied the extended atrributes, and even if you could, they would be wrong.
16:54 kshlm These xattrs are added only when data is added via the glusterfs client.
16:55 Joe630 zfs has even more attributes available (I think)
16:56 vpagan both servers have the same folder structure, just different files with unique filenames (if that makes a difference). I'm only bricking the zfs mounted partitions on each server to each other as a distribute volume
16:58 kshlm vpagan, again, did you rsync the remaining files directly into the brick folder?
16:58 vpagan yes
16:58 kshlm if that is the case, gluster doesn't know about the files you rsynced, so a rebalance wouldn't do anything.
16:59 vpagan what about the ones that were rsync'd?
16:59 vpagan weren't*
16:59 kshlm those are already in the correct place.
17:00 Joe630 vpagan:  I would mount the volume localally then just move them in there.
17:00 kshlm those files were put into brick1 by gluster, so they will not move out.
17:00 Joe630 that will take almost no time
17:00 Joe630 not the ones you rync'd in.
17:00 Joe630 those exist in one place - in the xfs filesystem on the locak cluster server.
17:01 vpagan kshlm, I did a remove-brick, which only moved some of the files, shouldn't those be moved back during the rebalance?
17:01 Joe630 files written to a brick outside of glusterfs daemon do not exist.
17:01 vpagan It was a few GB, but after adding the brick back and doing a rebalance, I only received 600mb
17:02 vpagan Joe630, I understand about those rsync'd files not going back now. But what about the ones that were put there during remove-brick from brick2?
17:02 vpagan I thought they would return after an add-brick then a rebalance
17:03 Joe630 i don't understand - you did a remove brick and it added files?
17:04 kshlm Joe630, a remove-brick would move files out of the brick being removed.
17:04 vpagan I had brick1 & brick2. I remove-brick brick2 which moves the files from brick2 to brick1 (right?). Then remove-brick commit (to remove brick2 from the vol). I saw not all the files went from brick2 to brick one, so I did an rsync
17:04 kshlm vpagan is talking about the file remove-brick moved from brick2 to brick1
17:05 vpagan then I brought back a new brick, did a rebalance, and did not get the full data
17:06 kshlm vpagan, you should still be able to recover from your present situation.
17:06 kshlm you still have your data, it's just that we need to bring it to glusters notice.
17:07 vpagan ok, how would I do that?
17:07 kshlm unfortunately, I don't have an exact answer for you.
17:08 kshlm You could run a 'ls -lr' or a 'find .' on a mount point. this should get gluster to identify all the files.
17:08 kshlm I'm not 100% sure this will work.
17:09 kshlm I've reached the limit of my knowledge on rebalance and distribute,
17:09 vpagan just so I understand you, anything I add into a gluster brick without using gluster, won't show up in the volume?
17:09 kshlm yes.
17:09 kshlm you might get them to show up in a ls
17:10 vpagan would a rebalance fix-layout force gluster to check?
17:10 vpagan or is that the same as a normal rebalance
17:10 kshlm on the mount point, but until you actually touch the file from a gluster mount, its invisible to gluster.
17:10 vpagan interesting
17:10 kshlm a fix-layout would just rewrite the directory layouts.
17:11 vpagan now, is there any reason why on a remove-brick it would say complete, but still have files left over?
17:11 kshlm please do a 'ls -lR' or a 'find . | stat' on a mount point. And then run a rebalance.
17:11 kshlm this should fix things.
17:12 kshlm remove-brick would leave behind files if there was no space availble in the rest of the volume.
17:12 kshlm since you had enough space, it shouldn't have left any behind.
17:13 vpagan I'll have to look into that then
17:13 kshlm by any chance, did you do the same thing for brick1? ie. remove it, rsync, expand and add-brick?
17:13 vpagan is a ls -lR > /dev/null ok since I have millions of files
17:14 vpagan I don't want to spam output
17:14 vpagan and it is possible I did the same to brick1 during my experimentation
17:14 kshlm will be fine. you just want to gluster to access a file.
17:14 kshlm :) so that might be why remove-brick on brick2 left some files behind
17:15 zerick joined #gluster
17:15 vpagan good to know
17:15 vpagan ok, my ls is done. Should I be rebalancing from a mountponit? I've always done it from one of the bricks
17:15 kanagaraj joined #gluster
17:16 kshlm rebalance doesn't depend on any mount. gluster will start a seperate process for doing the rebalance.
17:17 vpagan so I don't have to do a 'gluster volume rebalance'?
17:17 kshlm you have to do that.
17:17 Matthaeus joined #gluster
17:18 kshlm that command will start the rebalance process.
17:18 vpagan files are moving!
17:19 kshlm awesome.
17:19 rwheeler joined #gluster
17:20 vpagan kshlm, Joe630, thanks
17:20 vpagan one more question before I go though
17:20 nightwalk joined #gluster
17:20 kshlm sure.
17:20 seapasulli joined #gluster
17:20 vpagan I was told that while removing a brick, if a file is in use, gluster waits until the file is unlocked in order to complete the remove/rebalance
17:21 layer3switch joined #gluster
17:21 vpagan is there a timeout on that?
17:22 kshlm IIRC, rebalance does open file migration nowadays
17:22 vpagan great
17:23 vpagan thanks again
17:24 flrichar joined #gluster
17:25 jurrien joined #gluster
17:26 clutchk left #gluster
17:32 gmcwhistler joined #gluster
17:33 [o__o] joined #gluster
17:34 failshell when i run gluster volume geo-replication status --xml, i get that output: https://gist.github.com/failshell/ea50feb4ef5354d12fdf im guessing that's because that command didnt get the XML output implemented yet?
17:34 glusterbot Title: gist:ea50feb4ef5354d12fdf (at gist.github.com)
17:39 kshlm failshell, you guessed right.
17:40 failshell i find it odd that someone would spend time implementing XML output but not on all commands
17:42 kshlm well geo-rep was left out because the output it gives isn't controlled by glusterd.
17:42 kshlm for every other command glusterd gives the data.
17:43 kshlm but for geo-rep glusterd just forwards it to the geo-rep process,
17:43 failshell that's too bad. im writing a plugin for sensu. would of made my life easier than parsing a huge chunk of text
17:43 kshlm i implemented the xml output changes, and only did it for the commands glusterd answered directly to.
17:43 failshell ok
17:44 kshlm geo-rep is implemented in python, and I didn't feel like messing around in that portion of the code at the time.
17:44 kshlm though since it is in python, it should be easier to get it done.
17:44 failshell well, id implement it in json, but that's just a matter of personal preferences
17:45 kshlm I'd say json is better as well.
17:45 failshell easier to read by a human at least
17:46 kshlm It's just that the requirements I was given requested for xml output explicitly.
17:46 failshell gotcha
17:46 failshell do you work for RH by any chance?
17:46 kshlm Yup :)
17:47 failshell we use rhs
17:50 robos joined #gluster
17:53 chirino joined #gluster
17:55 JoeJulian +1 for json
18:05 nightwalk joined #gluster
18:14 FarbrorLeon joined #gluster
18:14 monotek semiosis?
18:23 chirino_m joined #gluster
18:25 primechuck joined #gluster
18:28 primechuck Are there any known issues or gotchas running Fuse+Bricks+KVM+MD Raid 5 on the same machine?  It looks like the system is being loaded up with kernel interupts from fuse/glusterd when the io load goes up to the point where anything requiring CPU time slows to a crawl.
18:38 robos joined #gluster
18:43 semiosis monotek: ?
18:43 monotek hi :-)
18:43 monotek i just tried your glusterfs-qemu ppa...
18:44 semiosis i need to update it with the new version in trusty
18:44 monotek in trusty... qemu-img is working for me but i dont get my libvirt vm started....
18:44 monotek yes, apt-pinned your repos so it works...
18:45 monotek do you have a working libvirt vm xml?
18:46 monotek i also was not able to add gluster pool:
18:46 monotek 2014-03-18 16:35:16.715+0000: 21748: error : virStorageBackendForType:1163 : internal error: missing backend for pool type 10 (gluster)
18:46 monotek 2014-03-18 16:35:16.715+0000: 21748: error : storageDriverAutostart:87 : Missing backend 10
18:46 jbustos joined #gluster
18:46 monotek libvirt is 1.2.2 so it should work, or not?
18:48 monotek at teh moment i use this in my libvirt vm xml:
18:48 monotek <disk type='network' device='disk'>
18:48 monotek <driver name='qemu' type='raw'/>
18:48 monotek <source protocol='gluster' name='/gv8/glustertest.img'>
18:48 monotek <host name='storage.domain' transport='tcp'/>
18:48 monotek </source>
18:48 monotek <target dev='vda' bus='virtio'/>
18:48 monotek </disk>
18:49 monotek when i start teh vm i get:
18:49 monotek 2014-03-19 18:48:40.000+0000: 22004: warning : qemuDomainObjBeginJobInternal:1119 : Cannot start job (modify, none) for domain glustertest; current job is (modify, none) owned by (22007, 0)
18:49 monotek 2014-03-19 18:48:40.000+0000: 22004: error : qemuDomainObjBeginJobInternal:1123 : Timed out during operation: cannot acquire state change lock
18:50 nightwalk joined #gluster
18:53 semiosis monotek: please use a ,,(paste) site for multiline pastes
18:53 glusterbot monotek: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
18:53 semiosis monotek: and i'm sure someone around here can help, but i've never used qemu-glusterfs (and haven't used libvirt/qemu in a while)
18:53 semiosis all i did was upload the package to the ppa, no idea how to use it
18:54 monotek sorry for not using paste services. thougt its short enough....
18:54 monotek so you didnt tested if libvirt will work with gluster/qemu?
18:55 monotek ok
18:55 monotek thanks for you work anyway :-)
18:56 semiosis i havent tested it but others have... i'm under the impression it works, tho idk exactly how to set it up
18:56 semiosis stick around i'm sure someone in here can help you
18:57 monotek ok, thanks... hope to find someone who is using it... have to leave for now....
18:58 ade_b joined #gluster
18:59 robo joined #gluster
19:00 nightwalk joined #gluster
19:11 discretestates joined #gluster
19:14 _dist joined #gluster
19:14 kkeithley joined #gluster
19:34 discretestates joined #gluster
19:35 nage joined #gluster
19:35 nage joined #gluster
19:51 qdk joined #gluster
19:53 colonha joined #gluster
19:53 colonha Hello everyone
19:54 glusterbot New news from resolvedglusterbugs: [Bug 960190] Gluster SHD and NFS do not start <https://bugzilla.redhat.com/show_bug.cgi?id=960190>
19:55 colonha Hello Joe
19:56 nightwalk joined #gluster
19:57 layer7switch joined #gluster
19:57 natgeorg joined #gluster
19:57 natgeorg joined #gluster
20:00 colonha ONe question
20:01 Matthaeus We'll hold you to that.
20:07 colonha I'm doing a rebalancing,  around 13TB, but the transfer speed is very slow, in 24 hours it copied just 170GB
20:07 colonha this is normal?
20:08 colonha can I speedup this operation?
20:09 Pavid7 joined #gluster
20:09 Matthaeus That's two questions.
20:09 Matthaeus Unfortunately, others in the channel would be more suited to answer that question than I would.
20:09 Matthaeus I haven't done much with rebalancing.
20:09 ThatGraemeGuy joined #gluster
20:10 colonha Ok Matthaeus, np.
20:10 colonha thx
20:10 lmickh joined #gluster
20:10 robo joined #gluster
20:10 Matthaeus Have you checked your process accounting and interface statistics to see if you're limited by hardware anywhere?
20:11 colonha The network transfer rate is very low
20:11 colonha Maximum 2Mbps
20:11 _dist colonha: I've done similar operations and seen the same thing. I've always believed, but never had confirmation of, that gluster throttles it's actions like rebalance to not tank the volume
20:11 robo joined #gluster
20:12 jiffe98 so I've upgraded 3 of my 4 nodes from 3.3 to 3.4 and after upgrading the last one, I'm getting State: Peer Rejected when it tries to connect to the other upgraded peers, logs complain about different versions and checksum mismatches
20:12 jiffe98 what's the best way to fix this?  looking in the volume directories I can see the different version numbers and checksums, I don't know if I can just modify those files and call it good?
20:13 _dist colonha: I've got a pipe that proveably can write 500-600MBytes/sec random i/o but gluster usually doesn't push beyond 50MBytes for operations like this. This is all just observation
20:13 colonha Ok, I saw a lot of performance options, but none related with rebalancing.
20:15 rfortier1 joined #gluster
20:15 _dist colonha: I've wondered about it too, but honestly something like a rebalance or adding bricks is so rare I don't mind that it takes days. Personally I haven't had to perform a rebalance on data that large, if I did I'd probably just rebuild the volume instead
20:15 colonha The more strange is that i have a 2 network interface (1Gb/inter) in each server configured with LACP (link aggregation control protrocol), but i got just 2Mbps
20:16 _dist colonha: what about an iperf test
20:16 colonha 956Mbps
20:16 colonha iperf test
20:16 _dist dd piped over netcat or ssh?
20:18 _dist realistically your network will probably only get around 100MBytes of real i/o with that iperf result, 20-30MMB on gluster rebalance sounds fine to me, 2MB/s does not
20:20 colonha 34.7 MB/
20:20 colonha 34.7 MB/s
20:20 colonha dd over ssh
20:21 _dist colonha: that might be because you're using default compression, use arcfour or something weaker
20:23 _dist local dds like "dd if=/dev/zero of=./test.img bs=1M count=2048 fdatasync" should also be tested
20:26 robo joined #gluster
20:27 colonha 76.9MB/s
20:27 colonha with arcfour
20:32 _dist colonha: that makes sense, so it sounds like it's taking so long because of a network or storage issue. After that I'm at the same wall you are
20:34 colonha Reviewing my server i see always my disk busy to 100% by the gluster procces
20:36 robo joined #gluster
20:38 isaacabo joined #gluster
20:39 isaacabo hello guys
20:42 nightwalk joined #gluster
20:42 japuzzo joined #gluster
20:42 isaacabo 24hrs ago we added a new brick to our gluster, just 2 brick now, and did a rebalance
20:42 dbruhn colonha, what version of gluster?
20:42 isaacabo ohh ,sorry, im with colonha too
20:42 dbruhn What version of gluster?
20:43 isaacabo [root@store1 glusterfs]# gluster --version
20:43 isaacabo glusterfs 3.4.2 built on Jan  3 2014 12:38:06
20:43 robo joined #gluster
20:45 dbruhn what is the status of the rebalance?
20:45 isaacabo [root@store1 glusterfs]# gluster volume rebalance shares status
20:45 isaacabo Node Rebalanced-files          size       sc
20:45 isaacabo ---------      -----------   -----------   ------
20:45 isaacabo localhost           256991       170.3GB        7
20:45 isaacabo store2.office.mentel.com                1       40Bytes        9
20:46 mattappe_ joined #gluster
20:46 mattappe_ joined #gluster
20:48 isaacabo we really can't pinpoint the issue
20:49 isaacabo there is no CPU consumption
20:50 isaacabo only disk activity
20:50 isaacabo LVM | ore1-lv_data | busy     70% | read      14 | write     13 | avio 25.5 ms |
20:50 isaacabo DSK |          sda | busy     70% | read      15 | write     13 | avio 24.6 ms |
20:50 dbruhn You'll have to forgive me I am behind on versions here, I am running 3.3.1
20:51 dbruhn The rebalance checks every single file in the file system and redistributes it based on the hash table
20:51 dbruhn so it's not going to move files based on size
20:51 dbruhn also are you going from one brick to two?
20:51 isaacabo yes
20:52 isaacabo brick1 was 12Tb, brick2 is 8Tb
20:52 isaacabo *is
20:52 dbruhn How full is the original brick?
20:52 isaacabo 96%
20:53 isaacabo and gluster is on top of a LVM
20:53 robo joined #gluster
20:53 dbruhn Lots and lots of small files?
20:56 isaacabo FS is XFS, and it's a mix... we have small and huge files
20:57 isaacabo lets say 60-70%% small files and 30%-40% big files
20:57 dbruhn from what's it's rebalanced it is about 1511 files per 1GB of data
20:58 dbruhn Rebalance ops are a common complaint among most people that do them.
20:58 dbruhn Often times it suggested that you do a fix layout first, that way the newly added bricks can be written to, if your data changes over a lot eventually the bricks will balance out
20:59 dbruhn if your data doesn't change a lot, then you can run a normal rebalance after and let it run slowly in the background.
20:59 isaacabo mmm
21:00 dbruhn A couple things you should probably keep an eye on, is that gluster really doesn't like asymmetrical brick sizes, it's distribution operations are not based on fill level
21:00 isaacabo ok, thanks
21:01 dbruhn Is there any reason you are running a single 12TB brick?
21:01 isaacabo and since is rebalancing it now, can we start our normal operation?
21:01 dbruhn Rebalancing happens in the background so the volume will continue to function
21:01 dbruhn it's just slow
21:02 isaacabo i'm new at the company, the just use the whole storage as one brick
21:02 isaacabo *they
21:02 isaacabo ok, tyvm
21:03 mattapperson joined #gluster
21:03 fraggeln joined #gluster
21:04 robo joined #gluster
21:04 isaacabo perhaps, we can add another 8TB brick, then remove the 12TB brick, and format it for another 8Tb brick
21:08 wrale joined #gluster
21:10 wrale For the life of me, I can't seem to figure out why on one of six hosts, the gluster NFS service isn't starting.. nfs.log there says "Could not register with portmap".. (On Fedora19) rpbbind.service is started.. Listening 0.0.0.0 on 111.. :(
21:10 wrale selinux is disabled
21:12 Pavid7 joined #gluster
21:12 robo joined #gluster
21:12 dbruhn wrale, disable the local NFS server
21:13 Joe630 wrale: gluster *is* the NFS server.
21:13 badone joined #gluster
21:15 wrale The local nfs server is not enabled, but I'll triple check.. When i said nfs.log, i mean /var/log/glusterfs/nfs.log
21:15 badone_ joined #gluster
21:15 Joe630 wrale: that is the same error i got when i tried to start nfs once
21:15 Joe630 before i found "the way"™
21:16 sroy_ joined #gluster
21:20 yinyin joined #gluster
21:22 failshel_ joined #gluster
21:31 nightwalk joined #gluster
21:32 robo joined #gluster
21:34 wrale No nfs server here.. nfs.target was disabled, however.. not sure what that's about..
21:35 wrale still no NFS Server on localhost=Y when doing gluster volume status
21:41 robo joined #gluster
21:43 fidevo joined #gluster
21:49 zerick joined #gluster
21:51 tdasilva left #gluster
21:59 seapasulli joined #gluster
22:02 ctria joined #gluster
22:08 robo joined #gluster
22:09 gdubreui joined #gluster
22:10 thefiguras joined #gluster
22:17 nightwalk joined #gluster
22:17 robo joined #gluster
22:23 rpowell1 joined #gluster
22:28 yinyin joined #gluster
22:31 robo joined #gluster
22:35 jiffe98 so I've upgraded 3 of my 4 nodes from 3.3 to 3.4 and after upgrading the last one, I'm getting State: Peer Rejected when it tries to connect to the other upgraded peers, logs complain about different versions and checksum mismatches
22:36 jiffe98 what's the best way to fix this?  looking in the volume directories I can see the different version numbers and checksums, I don't know if I can just modify those files and call it good?
22:36 dbruhn restart all the services include the glisterfs and glusterd and it will work
22:37 dbruhn The upgrade doesn't stop all of the services, so it's still reporting the earlier version
22:38 jiffe98 restart all the services on all machines?
22:38 jiffe98 or just the peer that is being rejected?  I killed all the gluster processes before I did each upgrade and reboot the machine
22:40 dbruhn if you rebooted all the peers that would work, but they all need the services restarted
22:41 jiffe98 there is only one node left to upgrade
22:42 dbruhn Yep upgrade the last one, restart the services, and you should be good to go
22:43 jiffe98 I'm thinking there's more to it than that but I'll give it a shot tonight
22:43 dbruhn the upgrade doesn't stop the glusterfs service, so you have clients trying to connect to older versions
22:44 dbruhn even though it's been ugpraded
22:44 dbruhn s/ugpraded/upgraded/
22:44 glusterbot What dbruhn meant to say was: even though it's been upgraded
22:44 dbruhn so by restarting the service, it loads the new version, and things become happy again
22:45 jiffe98 all 3 machines that have been upgraded have had their services restarted but one of them is being rejected by the other upgraded peers
22:45 dbruhn what services have been restarted?
22:46 jiffe98 gluster*
22:46 dbruhn which of the 4 peers is being rejected?
22:47 jiffe98 the last one I upgraded, I have gs1-gs4, gs3 and gs4 are upgraded and connected to eachother, gs1 is upgraded but being rejected
22:48 dbruhn I am assuming your volume is offline at this point?
22:48 jiffe98 no, clients connect to everything fine
22:48 dbruhn and what is your volume info?
22:49 jiffe98 http://nsab.us/public/glusterinfo
22:50 dbruhn why are you pausing on upgrading gs2?
22:51 jiffe98 because of the gs1 issue and it was running late and I don't want to break anything during the day
22:51 dbruhn makes sense, are you running iptables or any firewall between the servers?
22:51 dbruhn the ports changed from 3.3.x to 3.4.x
22:52 jiffe98 nope, firewall is outside the network
22:52 dbruhn distro?
22:52 jiffe98 ubuntu 12.04
22:52 jiffe98 I'm installing from sourcde
22:53 dbruhn well you can muck with your volume files if you want, but I would suggest backing them up first.
22:53 dbruhn There is an ubuntu repo so you don't need to compile from source
22:53 dbruhn it's maintained by semiosis
22:54 jiffe98 I probably can now, I used to have custom build options
22:55 Elico joined #gluster
22:56 jiffe98 any idea what the version inside /var/lib/glusterd/vols/[volname]/info is?
22:57 dbruhn no idea
22:58 dbruhn mine at 3.3.2 says 22
22:59 jiffe98 wondering if its an incremental version changing each time the volume changes
22:59 jiffe98 in which case it probably doesn't matter what it is so long as its the same across peers
23:04 nightwalk joined #gluster
23:20 B21956 joined #gluster
23:24 nightwalk joined #gluster
23:32 discretestates joined #gluster
23:35 neurodrone__ joined #gluster
23:36 fsimonce joined #gluster
23:39 Pavid7 joined #gluster
23:42 mattappe_ joined #gluster
23:43 nightwalk joined #gluster
23:59 chirino joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary