Camelia, the Perl 6 bug

IRC log for #gluster, 2013-01-29

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 chirino joined #gluster
00:16 melanor9 joined #gluster
00:33 yinyin joined #gluster
00:41 zhashuyu joined #gluster
01:29 bauruine joined #gluster
01:37 kevein joined #gluster
01:48 mohankumar joined #gluster
01:50 melanor9 joined #gluster
02:25 melanor9 joined #gluster
02:30 bharata joined #gluster
02:48 tomsve joined #gluster
02:57 hagarth joined #gluster
02:59 overclk joined #gluster
03:22 bulde joined #gluster
03:36 sgowda joined #gluster
03:42 sripathi joined #gluster
03:44 isomorphic joined #gluster
03:57 cgkades joined #gluster
04:00 cgkades i'm having issues running gluster, i'm using the install guide from the site. i've installed gluster-server and gluster-fuse (per the instructions). i had to install gluster-rdma because of an error in the log file was missing a .so. i'm still having an issue. http://pastebin.com/Pm7Lac34
04:00 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
04:00 polenta joined #gluster
04:01 cgkades also running centos 6.3
04:02 cgkades *hoping someone is awake that can help*
04:05 polenta cgkades, bit busy atm, but what is the question again? i've just got connected
04:06 sgowda joined #gluster
04:06 dustint joined #gluster
04:06 cgkades i'm having issues running gluster, i'm using the install guide from the site. i've installed gluster-server and gluster-fuse (per the instructions). i had to install gluster-rdma because of an error in the log file was missing a .so. i'm still having an issue. http://dpaste.org/fytG5/
04:06 glusterbot Title: dpaste.de: Snippet #217890 (at dpaste.org)
04:10 shylesh joined #gluster
04:10 polenta cgkades, do you have the link to the point of the install guide where you are getting stuck?
04:11 cgkades yeah, one sec
04:11 sahina joined #gluster
04:11 cgkades polenta: basically right after the yum install part http://www.gluster.org/community/d​ocumentation/index.php/QuickStart
04:11 glusterbot <http://goo.gl/OEzZn> (at www.gluster.org)
04:13 cgkades polenta: gluster wont run properly, and i cannot do the peer probe
04:14 polenta cgkades, I'll have a look later once a finish a few stuffs
04:15 cgkades polenta: no problem. i appreciate any help
04:16 cgkades polenta: it's complaining about an IB device. but i dont see that in the documentation, and i have no idea what it stands for
04:17 JoeJulian You didn't have to install gluster-rdma. That wasn't an error, that simply said that since the .so wasn't installed, you wouldn't have rdma support.
04:17 polenta cgkades, you even didn't create the volume yet, did you?
04:17 cgkades i followed the instructions in the guide
04:17 cgkades JoeJulian: i installed gluster-rdma, that solved the .so issue
04:18 JoeJulian No, it didn't solve the "issue" because there wasn't one. It did remove that message from the log though.
04:18 lala joined #gluster
04:18 JoeJulian Next: "[rdma.c:4993:init] 0-rdma.management: Failed to initialize IB Device". Do you have any infiniband devices?
04:18 cgkades JoeJulian: sorry, missread your comment
04:19 cgkades JoeJulian: no, i dont have an IB device.
04:19 JoeJulian Maybe we should approach this in a different way. What's not working?
04:19 cgkades JoeJulian: gluster peer probe <ip>
04:20 cgkades also if i do a service glusterd status, i don't see any of the informaiton that the guide shows
04:20 JoeJulian Is glusterd running on all your peers?
04:20 cgkades i have two peers, and yes
04:20 JoeJulian No firewalls?
04:21 cgkades i found a different guide on the site, and it's showing more steps before the peer probe
04:21 cgkades no, i disabled it
04:21 JoeJulian So what happens when you peer probe?
04:21 JoeJulian Oh, and did you clone your servers?
04:21 cgkades i'm bringing the other peer back online right now, and i'll paste the exact error
04:22 cgkades yeah i cloned them
04:22 JoeJulian @cloned servers
04:22 glusterbot JoeJulian: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
04:22 cgkades i'll check that out
04:24 cgkades gluster peer status returns "no peers present"
04:25 cgkades shoot.. i have the networking set up wrong so i could go online. let me fix that i'll be right back
04:26 melanor9 joined #gluster
04:27 lala joined #gluster
04:30 cgkades_ joined #gluster
04:31 cgkades_ does the line in the log about not being able to set up a listener mean anything?
04:34 cgkades_ well, the probe seems to work now
04:36 cgkades_ @clone
04:36 glusterbot cgkades_: I do not know about 'clone', but I do know about these similar topics: 'cloned servers'
04:36 cgkades_ @cloned servers
04:36 glusterbot cgkades_: Check that your peers have different UUIDs ('gluster peer status' on both). The uuid is saved in /var/lib/glusterfs/glusterd.info - that file should not exist before starting glusterd the first time. It's a common issue when servers are cloned. You can delete the /var/lib/glusterfs/peers/<uuid> file and /var/lib/glusterfs/glusterd.info, restart glusterd and peer-probe again.
04:39 cgkades_ cool seems to be working now, thanks JoeJulian
04:42 shireesh joined #gluster
04:45 overclk joined #gluster
04:49 rastar joined #gluster
04:53 lala_ joined #gluster
05:05 melanor9 joined #gluster
05:10 vpshastry joined #gluster
05:16 srhudli joined #gluster
05:20 eightyeight joined #gluster
05:20 ramkrsna joined #gluster
05:25 deepakcs joined #gluster
05:30 shireesh joined #gluster
05:30 overclk joined #gluster
05:33 raghu joined #gluster
05:34 sgowda joined #gluster
05:50 Humble joined #gluster
05:52 Humble joined #gluster
05:53 jmara joined #gluster
06:06 bulde raghaven1rabhat: ping
06:11 tru_tru joined #gluster
06:20 jmara joined #gluster
06:40 chirino joined #gluster
06:52 ngoswami joined #gluster
06:52 jmara joined #gluster
07:01 rgustafs joined #gluster
07:09 ekuric joined #gluster
07:23 puebele joined #gluster
07:26 jtux joined #gluster
07:39 guigui3 joined #gluster
07:40 isomorphic joined #gluster
07:41 puebele joined #gluster
07:45 glusterbot New news from newglusterbugs: [Bug 890618] misleading return values of some functions. <http://goo.gl/WsVnD>
07:50 Crypticfortune joined #gluster
07:55 shireesh joined #gluster
08:00 ctria joined #gluster
08:06 jtux joined #gluster
08:12 Crypticfortune Hi. I'm testing out gluster, and just tried adding a new brick and ran rebalance fix-layout start. status says it's been on "rebalance step 1" for about 5 minutes now. is it dead?
08:13 Crypticfortune also, the other node (there's only 2) says "rebalance not started" which makes me suspicious
08:14 Nevan joined #gluster
08:17 Crypticfortune oh, "rebalance" alone seems to be a synonym for "do fix-layout then do migrate-data"? So I guess there's 2 steps?
08:23 andreask joined #gluster
08:27 tjikkun_work joined #gluster
08:31 Joda joined #gluster
08:39 DaveS_ joined #gluster
08:45 lh joined #gluster
08:45 lh joined #gluster
08:49 melanor9 joined #gluster
08:55 bulde joined #gluster
08:58 dobber joined #gluster
08:59 sgowda joined #gluster
09:00 w3lly joined #gluster
09:03 srhudli joined #gluster
09:03 shireesh joined #gluster
09:14 overclk joined #gluster
09:16 gbrand_ joined #gluster
09:18 grzany joined #gluster
09:26 x4rlos Hi all. Can i clear up a few things please with someone.
09:26 x4rlos Firstly, I have set up a normal 2 brick replication between 2 machines at two sites.
09:27 x4rlos Im looking to use to keep my database archive files.
09:27 x4rlos So i have: mount -t glusterfs server1:/archive-name ./srv/local_archive_mount
09:28 x4rlos But this will sync to both server1 and server2 in the brick "farm" right?
09:28 x4rlos So if server1 dies, it will still sync with server2?
09:30 melanor91 joined #gluster
09:31 jtux joined #gluster
09:33 partner x4rlos: yes
09:33 melanor92 joined #gluster
09:33 partner and once the server1 comes back online, with a recent glusterfs, it will self-heal and update the files on the server1 brick
09:33 Staples84 joined #gluster
09:37 Humble joined #gluster
09:38 melanor9 joined #gluster
09:46 x4rlos Thanks. I have tested this already, but for some reason, i felt the need to check :-)
09:54 melanor91 joined #gluster
09:58 gbrand__ joined #gluster
09:59 x4rlos Is there any suggestion to the owner/group of the mountpoint the gluster volumes are on? In the past i just assign 777 to them, and then let gluster decide the permissions. Is there a preferred alternative method to this?
10:01 Azrael808 joined #gluster
10:07 x4rlos User manual doesn't _seem_ to suggest a preferred method.
10:09 sripathi joined #gluster
10:10 rastar joined #gluster
10:18 ramkrsna joined #gluster
10:26 Humble joined #gluster
10:29 shireesh joined #gluster
10:40 Norky joined #gluster
10:42 partner x4rlos: one suggestion is to make your brick mode 700 - nobody is supposed to go there straight
10:43 deepakcs joined #gluster
10:43 partner x4rlos: and owned by root as gluster runs under it
10:44 partner darn, sorry, misread again
10:45 partner the volume mountpoint doesn't differ from any other mounts really, make the permissions as needed
10:54 NuxRo howdy partner
10:55 NuxRo happy with the answer on the write failures on distirbuted volumes? :)
10:56 shireesh joined #gluster
10:56 ngoswami_ joined #gluster
11:00 hateya joined #gluster
11:06 x4rlos partner: thanks for this.
11:07 x4rlos when you say make my brick mode 700 - how do you mean? I was going to implement the restrictions on the gluster set options.
11:08 x4rlos I have the postgres user that will want to write to these gluster mountpoints.
11:08 Humble joined #gluster
11:09 x4rlos So i should 1) chmod 777 on the servers directly .. 2) gluster volume set gv0 auth.allow 192.168.111.30,192.168.111.31
11:09 manik joined #gluster
11:10 w3lly joined #gluster
11:13 x4rlos Can i tell gluster which ips to have rw and which to have ro? (as nfs does)
11:14 ndevos x4rlos: no, I'm pretty sure you cant, recent glusterfs-client versions can mount read-only though
11:16 x4rlos no probs. Thanks :-)
11:34 manik joined #gluster
11:37 Humble joined #gluster
11:45 edward1 joined #gluster
11:51 vpshastry joined #gluster
11:51 partner NuxRo: umm what you mean?
11:59 partner ok, read the answers. however, my point was to address that problem because people _assumed_ it would work somehow more wisely and end up having issues described
12:04 andreask joined #gluster
12:05 sashko joined #gluster
12:08 nueces joined #gluster
12:10 kshlm|AF1 joined #gluster
12:13 shireesh joined #gluster
12:23 nueces joined #gluster
12:28 luis_alen joined #gluster
12:28 Norky joined #gluster
12:29 vpshastry joined #gluster
12:30 mohankumar joined #gluster
12:31 luis_alen Hello, guys. Is it possible to rename volumes on version 3.3? If yes, is there any doc? 3.1 has a "gluster volume rename", but I don't see it on 3.3
12:37 kkeithley1 joined #gluster
12:38 polfilm joined #gluster
12:46 glusterbot New news from newglusterbugs: [Bug 905425] 3.4.0qa7 doesn't show correct volume size when quotas used <http://goo.gl/WdvQA>
12:50 tryggvil joined #gluster
12:56 ngoswami_ joined #gluster
12:56 shireesh joined #gluster
12:57 mohankumar joined #gluster
13:19 w3lly joined #gluster
13:21 mohankumar joined #gluster
13:23 partner "only way to get stats out of glusterfs is to add a debug translator, set an extended attribute, and scan the server's logfile for output" - anything fresh out in the field of monitoring?-) namely nagios and munin at least for starters
13:30 hagarth joined #gluster
13:30 partner but yes, will do some more googling around or come up with my own
13:31 dustint joined #gluster
13:40 mohankumar joined #gluster
13:55 mohankumar joined #gluster
14:00 melanor9 joined #gluster
14:01 ekuric left #gluster
14:03 NeatBasis joined #gluster
14:03 mohankumar joined #gluster
14:08 tomsve joined #gluster
14:12 x4rlos partner: I was just looking at monitoring too.
14:12 x4rlos what you got so far?
14:25 stat1x joined #gluster
14:31 mohankumar joined #gluster
14:35 chouchins joined #gluster
14:39 luckybambu joined #gluster
14:40 lala joined #gluster
14:47 Staples84 joined #gluster
14:53 rastar joined #gluster
14:57 deepakcs joined #gluster
14:59 lkoranda_ joined #gluster
15:02 stopbit joined #gluster
15:03 plarsen joined #gluster
15:06 bennyturns joined #gluster
15:08 mohankumar joined #gluster
15:09 rwheeler joined #gluster
15:14 kshlm joined #gluster
15:17 partner x4rlos: i recall not finding anything immediately usable. the amount of glue on the found ones exceeds my tolerance, i will do another sweep around the intttternet later on
15:18 mohankumar joined #gluster
15:19 x4rlos partner: Any way of using the gluster client on the monitoring node to try and probe for info via peer status without being attached to any bricks?
15:19 x4rlos alternatively, there;s always ssh commands i guess.
15:19 x4rlos or exporting via custom snmp (yuk)
15:21 wushudoin joined #gluster
15:21 jbrooks joined #gluster
15:24 bugs_ joined #gluster
15:25 vpshastry joined #gluster
15:26 partner x4rlos: i have all the means in place to do whichever approach, probably mostly on peers as they see all the details
15:31 ninkotech_ joined #gluster
15:34 x4rlos partner: githubb'd?
15:35 neofob joined #gluster
15:36 tryggvil joined #gluster
15:40 sebastian__ joined #gluster
15:41 hagarth joined #gluster
15:42 chirino joined #gluster
15:44 ctria joined #gluster
15:44 ekuric joined #gluster
15:45 sebastian__ Hi, can anyone advise if Gluster client has any kind of auto failover feature?
15:46 daMaestro joined #gluster
15:47 mohankumar joined #gluster
15:51 Humble joined #gluster
15:51 elyograg sebastian__: the fuse client talks to all your bricks, so if you have replication in place and part of it goes down, it should continue to work with the bricks that are still up.  the NFS server capability has no built-in failover, you have to provide that.
15:55 sebastian__ elyograg: Thx for reply. I was doing a simple test with a while loop writing to a file and then I powered off the node that the clinet was mounted on. I was monitoring this file on a second node and the client stooped writing to it and just hung
15:58 elyograg sebastian__: if you used the NFS mount (didn't specify -t glusterfs when mounting) then that makes sense. If you did, then it would hang for about 42 seconds, then keep going.
15:59 sebastian__ I used the native client ( mount.glusterfs node1:datavol /mnt/glusterfs/)
16:00 nueces_ joined #gluster
16:00 sebastian__ Is it possible to change the time out value (this ~42 sec)
16:01 nueces joined #gluster
16:01 elyograg the fuse mount (-t glusterfs) only uses the hostname that you give it during mount, once connected it knows about the entire volume.  If it didn't recover, perhaps you have connectivity problems from the client to the other server(s) by the names used in the volume creation.
16:08 sebastian__ elyograg: Thanks for your help. I will keep testing
16:14 atrius joined #gluster
16:18 aliguori joined #gluster
16:28 chirino joined #gluster
16:37 mohankumar joined #gluster
16:39 samppah @latest
16:39 glusterbot samppah: The latest version is available at http://goo.gl/zO0Fa . There is a .repo file for yum or see @ppa for ubuntu.
16:40 samppah @qa
16:40 samppah @beta
16:40 samppah mhm
16:40 ninkotech_ joined #gluster
16:44 semiosis @qa repo
16:44 semiosis @qa releases
16:44 glusterbot semiosis: The QA releases are available at http://bits.gluster.com/pub/gluster/glusterfs/ -- RPMs in the version folders and source archives for all versions under src/
16:44 semiosis there it is
16:45 semiosis samppah: ,,(qa releases)
16:46 glusterbot samppah: The QA releases are available at http://bits.gluster.com/pub/gluster/glusterfs/ -- RPMs in the version folders and source archives for all versions under src/
16:46 samppah semiosis: thanks :)
16:46 semiosis yw
16:57 mohankumar joined #gluster
17:01 sashko joined #gluster
17:11 tryggvil joined #gluster
17:33 Mo__ joined #gluster
17:41 amccloud joined #gluster
17:42 chirino joined #gluster
17:52 polfilm joined #gluster
17:55 ctria joined #gluster
17:59 nueces joined #gluster
18:09 y4m4 joined #gluster
18:18 cable2999 joined #gluster
18:23 amccloud Are there any docs on creating volumes and etc from a config file?
18:23 amccloud pools and volumes
18:31 amccloud I'm using chef for configuration management so I'd like to stay away from scripting the cli.
18:33 semiosis there's a couple ,,(puppet) modules
18:33 glusterbot (#1) https://github.com/semiosis/puppet-gluster, or (#2) https://github.com/purpleidea/puppet-gluster
18:34 semiosis mine avoids actually probing peers & creating volumes
18:34 semiosis purpleidea's calls the gluster command iirc
18:36 amccloud nice i
18:36 amccloud i'll check it out
18:36 semiosis good luck, administering glusterfs with config mgmt is harder than it seems at first
18:37 semiosis since gluster cli already does both config mgmt and server orchestration
18:37 amccloud Oh? What type of problems have you ran into?
18:40 semiosis defining the problem :)
18:40 partner i was thinking that for a moment but some things are just too difficult to handle via configuration management.. or one could just manage the vol files and then command via cli to do some actions but at least initially thinking gets too complicated
18:40 semiosis messing with vol files is deprecated (not recommended)
18:40 partner which one?-)
18:41 semiosis and there's currently no way to mix gluster cli & manual vol file editing
18:41 partner that is an issue then
18:41 semiosis gluster cli will blow away any changes you've made to vol files
18:41 amccloud I see
18:41 partner how the cli gets the config then if not from vol file, there must be some source..
18:41 semiosis and if you don't use gluster cli then you lose online volume changes
18:42 partner due to all above its why i decided not to even attempt
18:42 partner its just easier to document it all down :)
18:42 amccloud okay cli it is!
18:42 semiosis partner: cli persist state to files, but the actual state is running in process
18:42 partner rgr
18:44 semiosis amccloud: defining the problem is hard.  my essential need was monitoring & ensuring system was set up to support glusterfs... having bricks in fstab, nagios set up to monitor, etc
18:44 semiosis amccloud: so i set all that up with puppet then used gluster cli to probe peers & volumes
18:44 semiosis s/&/& create/
18:45 glusterbot What semiosis meant to say was: (histsearch <an alias, at least 1 argument>) -- Alias for "last --from [echo semiosis] --regexp "/^(?!s([^A-Za-z0-9\}\]\)\>\{\[\​(\<\\\]).+\\1.+[ig]*).*$1.*/@1" --in [echo #gluster]".
18:45 semiosis yeah no
18:45 partner heh
18:45 partner yeah one could do it partially too, i am bringing cfengine here so we have all the most popular ones in use :)
18:46 semiosis cool
18:46 partner one of the most important things in "cf management" is IMO the version history, i'd rather call it change management
18:47 semiosis more ambitious people have suggested using config management to automatically replace a failed disk or server, or automatically scale out the cluster
18:47 partner for gluster that would be nice too
18:48 partner semiosis: its very much doable BUT :)
18:48 semiosis imho those goals are not practical at this point
18:49 semiosis partner: i would love to be proven wrong on that
18:49 partner it so much depends on the infra, how large it is and how much for example gluster node failures cause work for you, etc
18:49 partner semiosis: on what? sorry
18:50 semiosis i'd love to be proven wrong on my opinion that it's not practical to build an auto-repairing or auto-scaling glusterfs cluster
18:50 semiosis iow i'd love to see someone do it
18:50 semiosis but i dont think it can be done
18:50 partner anything can be done, that is not the issue
18:51 semiosis s/done/done practically/
18:51 glusterbot What semiosis meant to say was: but i dont think it can be done practically
18:51 partner i agree, practicality comes into picture quite fast
18:52 partner maybe.. haven't thought that far, trying to build systems so that i don't have to be running immediately, take into account that hw _will_ fail and so forth
18:53 partner my current issue is really to get monitoring into shape, i have no idea if everything is ok..
18:54 semiosis partner: there's a bunch of nagios checks in my puppet module that may help you
18:54 partner yeah, will check them through
18:55 semiosis that was written before 'gluster volume status' command was available though... theres more ways to monitor bricks now
18:57 partner w00t.. is the brick port number ever increasing number and not as documented (somewhere) from 24009 onwards brick count!?
18:57 semiosis ,,(ports)
18:57 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
18:58 semiosis partner: and even that's going to change in a future release
18:58 hattenator joined #gluster
18:58 partner right
18:59 partner well, that's why i'm here, you learn new things every day, thanks :)
19:00 partner not that i was hit by that, its just many examples out there give you small port ranges to which probably many will bump into while testing
19:00 melanor9 joined #gluster
19:00 partner or maybe others are more brave and go straight into production
19:01 flrichar joined #gluster
19:02 flrichar hello all
19:03 flrichar I haven't played with glusterfs yet, the details are all bouncing around in my head
19:04 flrichar but I have a question... if all bricks on hosts need to be the same size, any issue with putting the drives under lvm control, to ensure bricks (lvm volumes) are the same size?
19:04 semiosis it's more a best-practice recommendation than a need, but yes lvm is ok
19:05 flrichar yea and that was my struggle, if the "one disk per brick" vs "lvm volumes for bricks" clashed as best practices
19:05 semiosis glusterfs doesnt care what the underlying block devices are as long as they're formatted & mounted with a POSIX filesystem that supports xattrs
19:05 flrichar right, I'm gonna stick with xfs for now
19:05 semiosis xfs is recommended by glusterfs
19:06 semiosis specifically, inode size 512 to accomodate the xattrs
19:06 cable2999 joined #gluster
19:06 fixxxermet left #gluster
19:06 flrichar yup, cool.  so I can limit my bricks to 128g, I probably won't have files that large
19:07 flrichar and I can guess replication happens across hosts, not bricks
19:07 semiosis bricks
19:07 flrichar bricks seem like the lvm equivalent of extens
19:07 semiosis see ,,(brick naming)
19:07 glusterbot http://goo.gl/l3iIj
19:07 semiosis ,,(glossary)
19:07 glusterbot A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
19:07 semiosis brick is directory-on-a-server
19:08 chirino joined #gluster
19:08 flrichar hmm so if you say 2 servers, 4x brick per server... replicate 2x ... files could end up on a server twice?
19:08 partner (do people get mad here if one links to RH docs regarding the above questions, its all there with nice graphs and stuff?)
19:08 flrichar or if the 4 bricks are in different volumes...
19:09 semiosis partner: actually if you have links to good docs that would be appreciated
19:09 semiosis flrichar: glusterfs will warn you if you try to replicate between two bricks on the same server
19:09 flrichar cool
19:09 semiosis but yes it can be done
19:09 semiosis i do that often for testing
19:10 partner i just happen to have links readily open to answer above questions.. here we go, feel free to kick me..
19:10 partner https://access.redhat.com/knowledge/docs/en-US/​Red_Hat_Storage/2.0/html/Administration_Guide/c​hap-User_Guide-Setting_Volumes.html#idp8039568 - formatting
19:10 glusterbot <http://goo.gl/zlHPx> (at access.redhat.com)
19:10 flrichar yea I was thinking if there's brick0-3 per server (under lvm control) it would be ok, provided they're 4 gluster volumes
19:10 elyograg kkeithley1: have you got a new HOWTO for the new UFO that uses 1.7.4 and keystone?  I would need some minimal instructions on setting up keystone as well.
19:10 partner flrichar: https://access.redhat.com/knowledge/docs/en-US/​Red_Hat_Storage/2.0/html/Administration_Guide/s​ect-User_Guide-Setting_Volumes-Replicated.html
19:10 glusterbot <http://goo.gl/Zo2Tu> (at access.redhat.com)
19:11 elyograg or pointers to other instructions that would mesh with a new HOWTO.
19:11 flrichar awesome thanks
19:11 Ryan_Lane joined #gluster
19:11 partner flrichar: sorry, this one for many bricks ie. distributed replicated: https://access.redhat.com/knowledge/docs/en-US/​Red_Hat_Storage/2.0/html/Administration_Guide/s​ect-User_Guide-Setting_Volumes-Replicated.html
19:11 glusterbot <http://goo.gl/Zo2Tu> (at access.redhat.com)
19:11 Ryan_Lane semiosis: glusterfs package in ppa still doesn't replace old glusterfs package properly :(
19:12 semiosis Ryan_Lane: did you kill all gluster-related processes from the server before upgrading?
19:12 partner it shows you how the files are supposed to spread around to bricks (you can for example first do 2 brick replicate and then extend that to 4 brick (or more) distributed replica)
19:12 Ryan_Lane this is the client package
19:12 Ryan_Lane http://dpaste.com/898917/
19:12 glusterbot Title: dpaste: #898917 (at dpaste.com)
19:12 Ryan_Lane oh. do the client processes need to be killed too?
19:14 Ryan_Lane yeah. still get that error
19:14 semiosis Ryan_Lane: latest ppa package is glusterfs - 3.3.1-ubuntu1~precise5 but you're trying to install precise4
19:14 Ryan_Lane let me try doing an apt-get update again
19:15 Ryan_Lane seems to still be trying 4
19:15 Ryan_Lane no clue why
19:15 ninkotech_ joined #gluster
19:16 semiosis well that's the issue, the precise4 release didnt replace the old gluster.org package
19:16 andreask joined #gluster
19:21 Ryan_Lane ah. it's in my custom repo. good reason as to why it wouldn't get precise5
19:29 Ryan_Lane semiosis: works perfectly. sorry for the mixup ;)
19:29 semiosis \o/
19:30 semiosis no worries
19:30 theron joined #gluster
19:41 cicero /sbin/mount.glusterfs: according to mtab, GlusterFS is already mounted on /data
19:41 cicero eep. it's not mounted, and i just added it to /etc/fstab
19:41 cicero does anyone know what that means?
19:42 dberry joined #gluster
19:42 dberry joined #gluster
19:43 DaveS joined #gluster
19:43 dberry Does anyone have a tutorial on migrating from glusterfs 3.0.2 server to 3.3?
19:46 cicero hmm. actually it seems if i change the mountpoint to something else, it mounts
19:46 cicero so i wonder if that's a substring failure
19:47 cicero (this server has another glusterfs mount point /data2, and i can't mount /data from another gluster server)
19:47 DaveS__ joined #gluster
19:51 cable2999 joined #gluster
19:57 erik49 joined #gluster
20:15 nueces joined #gluster
20:22 melanor9 joined #gluster
20:23 aliguori joined #gluster
20:29 JoeJulian cicero: Check /proc/mounts
20:29 JoeJulian dberry: No tutorial. Just recreate your volume from the cli.
20:36 kombucha I'm getting "Transport endpoint is not connected" when I try to ls a distribute gfs volume mounted locally
20:36 kombucha haven't been able to discern what's going on with it
20:36 kombucha I was able to mount a different replica volume on this machine successfully, but getting this error with a single brick distribute volume
20:37 JoeJulian Sounds like the client crashed or is not connected to the bricks. Check the client log.
20:37 mnaser joined #gluster
20:37 JoeJulian Or the volume isn't started...
20:38 kombucha vol status says started!  :-)
20:38 kombucha one thing I noticed on the client was  0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
20:38 kombucha but that's just defaulting to the tcp socket right
20:38 JoeJulian normal
20:38 kombucha yes
20:38 JoeJulian yep
20:39 JoeJulian Feel free to [fd]paste your client log if you want me to take a look.
20:41 kombucha JoeJulian: does this indicate anything to you?
20:41 kombucha W [fuse-bridge.c:419:fuse_attr_cbk] 0-glusterfs-fuse: 10544: LOOKUP() / => -1 (Transport endpoint is not connected)
20:42 JoeJulian Yep, that says that the tcp connection is not established to any server.
20:44 kombucha right, except netstat on the server running gluster shows an established connection on 24007
20:45 kombucha any chance this has anything to do with the brick being defined with the hostname not IP? (ie if a dns lookup on the brick fails...) ?
20:46 kombucha And is there a doc page on how to remove a brick, and then re-add it by IP, not host name?
20:48 m0zes @ports
20:48 glusterbot m0zes: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
20:49 m0zes @hostnames
20:49 glusterbot m0zes: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
20:50 semiosis kombucha: use hostnames, if dns resolution fails, fix dns
20:50 semiosis just my opinion :)
20:50 kombucha semiosis: thanks, that was my thinking too.
20:51 kombucha and thanks m0zes for the botfacts
20:51 semiosis iirc the client only does dns lookups when it first starts up
20:51 semiosis i seem to remember having to unmount/remount clients when the CNAME of one of my servers changed to a diff IP
20:51 kombucha it seems to be mounting ok, and server shows an established connection, but attempting to ls the directory on the client returns the Transport endpoint not connected error
20:52 semiosis kombucha: check client log file, usually /var/log/glusterfs/client-mount-point.log
20:52 kombucha with a couple : getaddrinfo failed in the client logs
20:52 semiosis common causes of that are... 1) volume created but not started.  2) name resolution not set up correctly.  3) iptables
20:52 kombucha yeah that's the one with the getaddrinfo fails in it
20:53 semiosis pastie.org the log?
20:53 Mo__ joined #gluster
20:56 kombucha pasted 75 lines for context, but it's pretty much the same error repeated --> http://pastebin.com/9BkTyXfj
20:56 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
21:02 kombucha thanks, I'll change pastebin preferences
21:07 semiosis kombucha: do you have a 'search ...' line in resolv.conf?
21:08 semiosis if 'DNS resolution failed on host arche-uat1' means tried to look up host 'arche-uat1' then i can see why that would fail
21:08 kombucha no, just a nameserver
21:08 semiosis no domain
21:08 kombucha nope
21:08 semiosis kombucha: ,,(pasteinfo)
21:08 glusterbot kombucha: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:08 kombucha I have entries in the local host file, but it seems as if they're not being referenced
21:09 semiosis nsswitch?
21:10 kombucha http://dpaste.org/Fqm2p/
21:10 glusterbot Title: dpaste.de: Snippet #217961 (at dpaste.org)
21:10 kombucha and thanks glusterbot for the cleaner pastebin
21:10 kombucha that's when I had it stopped bt
21:10 kombucha btw
21:11 semiosis heh
21:11 semiosis if you run 'host arche-uat1' from the client machine does it resolve ok?
21:11 kombucha hang on to your hat, it's doing it
21:12 kombucha I fixed dns entries
21:12 kombucha so this was entirely a namespace problem
21:12 kombucha different members of the team use different conventions for host names
21:12 semiosis er not host
21:12 semiosis ping arche-uat1
21:12 kombucha well it's working now!
21:12 semiosis yay
21:13 kombucha host arche-uat1
21:13 kombucha Host arche-uat1 not found: 3(NXDOMAIN)
21:13 kombucha still, it is working.
21:13 semiosis yeah forgot that doesnt check /etc/hosts file just dns
21:13 kombucha yeah, I wasn't sure about that myself at first, only discovered that through trial/error
21:14 kombucha hey, thanks for the suggestions and encouragement   :::shakes fist at dns::::
21:14 semiosis yw
21:15 kombucha semiosis: while you're here, I am trying to resolve some contradictory information.  From the documentation, it appears that a client pointed at a replica cluster can failover if one of the bricks in the cluster goes down
21:15 kombucha however in conversation I was told that it doesn't failover to another brick in the cluster
21:15 semiosis ,,(mount server)
21:15 glusterbot (#1) The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds, or (#2) Learn more about the role played by the server specified on the mount command here: http://goo.gl/0EB1u
21:15 kombucha I think it was you I saw posting on a similar thread about athat
21:16 badone joined #gluster
21:16 kombucha right, that's almost a verbatim quote from the doc page I was referrring to
21:16 semiosis a winner is glusterbot
21:16 semiosis hehe
21:16 kombucha but the person who was telling me there was an issue with that works at RH on gfs, so I just wanted to get some veification!
21:17 semiosis gfs?
21:17 kombucha heh
21:17 kombucha ok that page is EXACTLY what I was looking for
21:17 kombucha it's a fuse vs nfs issue. that perfectly explains it. thanks again.
21:18 semiosis yw
21:18 kombucha Hoping to have a gluster event / camp / something this summer in NYC btw  :-)
21:18 * kombucha wanders off for a bit
21:19 semiosis johnmark: ^^^^
21:27 tryggvil joined #gluster
21:29 JoeJulian "<semiosis> i seem to remember having to unmount/remount clients when the CNAME of one of my servers changed to a diff IP" - Changed in 3.3
21:30 semiosis awesome, one more reason i should upgrade
21:31 nueces joined #gluster
21:32 partner 23:13 < semiosis> yeah forgot that doesnt check /etc/hosts file just dns
21:32 JoeJulian He's referring to the "host" command.
21:32 partner oh
21:32 partner true
21:33 partner i was wondering if it was bypassing all the usual channels.. my bad
21:33 JoeJulian I read that wrong the first time I read the scrollback myself.
21:34 partner i have hard time ever remembering which exact tool uses what ways to resolve..
21:34 partner though using hosts is as evil as it can get..
21:35 semiosis (OT) recently found out that dnsmasq can answer all A queries for a domain with the same IP
21:35 JoeJulian It would be nice if "host" had a switch that did work with /etc/hosts.
21:36 JoeJulian That's what the rfc requires, so that's good. :)
21:36 partner i think if anything uses hosts, everything should use it, otherwise it simply makes debugging very difficult
21:37 JoeJulian Except that "host" is part of bind, not nss
21:37 partner or rather NSS
21:38 JoeJulian It's supposed to be a tool for testing bind.
21:41 partner that doesn't leave much tools for testing what the host sees.. ping to the rescue :)
21:42 partner but i'm sure there's enough debate out there around the topic, use the ping luke
21:44 JoeJulian I'd prefer a 3-5 letter command that you can just add a hostname to and see what getaddrinfo sees. Though I could write a python 1-liner to do that, it would be nice if it were in a default package.
21:45 partner i'm not sure if i would trust it for python, it does some funky things with its resolvings..
21:46 mmakarczyk joined #gluster
21:46 JoeJulian It's just calling the glibc function.
21:46 partner i guess there are several ways of doing it but i've been witnessing how it sorts for example round-robin addresses and what not
21:47 partner but that might have been related to something.. urls or something, cannot recall, it was weird enough to only always hit the first boxes
21:51 Ryan_Lane this upgrade is going really poorly
21:51 Ryan_Lane almost all of my volumes show at minimum one brick down
21:51 Ryan_Lane most show three bricks down
21:52 cicero JoeJulian: confirmed that /proc/mounts doesn't have the mount point i'm trying to mount
21:53 cicero JoeJulian: but i'm pretty sure it's because i'm trying to mount a /data and i already have a glusterfs mnt point named /data2
21:53 cicero JoeJulian: because when i changed from /data to /lols, it worked :\
21:53 JoeJulian ~pasteinfo | cicero
21:53 glusterbot cicero: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:53 cicero nice piping
21:54 cicero sec
21:56 cicero JoeJulian: http://fpaste.org/NOBg/
21:56 glusterbot Title: Viewing Paste #271804 (at fpaste.org)
21:56 duerF joined #gluster
21:58 JoeJulian That's interesting. You don't have fs3 and fs1 as part of the same peer group (you don't have to for this, just curious)
21:58 cicero yeah i wasn't sure what that would do
21:58 JoeJulian I'll need the volume info from fs1 though
21:58 cicero actually.
21:58 cicero because they're bricks in different volumes
21:58 cicero sec
21:59 cicero updated: http://fpaste.org/D7Qw/
21:59 glusterbot Title: Viewing Paste #271805 (at fpaste.org)
22:00 JoeJulian Mmkay... it's not that...  grep data /etc/mtab (on fs3)
22:00 Ryan_Lane hooray. seems I'm having a full-fledged outage :(
22:00 cicero JoeJulian: same output as grepping /proc/mounts
22:00 cicero i mean just the one line for /data2
22:00 cicero but no /data
22:01 cicero if i have time i'll reproduce in a clean environment and then submit a bug report
22:01 JoeJulian Ugh, Ryan_Lane.
22:02 * JoeJulian rages too... "You don't exist, go away!" Thanks, Lennart.
22:04 Ryan_Lane it seems there's no way for me to stop the gluster server?
22:05 JoeJulian killall glusterfsd
22:05 JoeJulian cicero: You're right. It's a bug. "if grep -q " ${mount_point}.*fuse" $mounttab; then"
22:05 Ryan_Lane yeah, but glusterd seems to restart itself
22:05 Ryan_Lane after stopping it with upstart
22:06 cicero JoeJulian: nice -- is that from teh source?
22:06 JoeJulian cicero: Yes. mount.glusterfs
22:06 JoeJulian Ryan_Lane: There's nothing in glusterd (source) to do that.
22:06 JoeJulian Must be semiosis' fault.
22:06 Ryan_Lane well, it's probably upstart
22:06 * JoeJulian throws semiosis under the bus.
22:07 cicero :D
22:07 semiosis glusterbot: meh
22:07 glusterbot semiosis: I'm not happy about it either
22:07 JoeJulian It would have to be competing upstart requirements.
22:08 JoeJulian cicero: A (stupid) workaround would be to mount /data before /data2
22:09 cicero JoeJulian: yeah i'm gonna submit a patch
22:09 JoeJulian @hack
22:09 cicero but good call
22:09 glusterbot JoeJulian: The Development Work Flow is at http://goo.gl/ynw7f
22:09 cicero kthx
22:11 Ryan_Lane start: Job is already running: glusterfs-server
22:12 Ryan_Lane there's no glusterd process
22:12 Ryan_Lane this is after a reboot
22:12 amccloud joined #gluster
22:12 Ryan_Lane no pid file
22:13 cicero this won't help, and i apologize, but i hate upstart
22:14 cicero If you try to start a job that is already running and which does not specify the instance stanza, you will get the following error:
22:14 cicero start: Job is already running: myjob
22:14 cicero no idea what this instance stanza is
22:15 semiosis supporting multiple instances of the same job
22:15 Ryan_Lane seems the old package wasn't removed
22:15 cicero ah
22:15 semiosis doesnt apply to gluster
22:15 Ryan_Lane let's see if rebooting helps
22:15 Ryan_Lane after removing the old package
22:15 amccloud joined #gluster
22:16 semiosis upstart is new and different and misunderstood, therefore hated by lots of people. seen that before.
22:17 Ryan_Lane I hate it for this exact reason
22:17 Ryan_Lane it will get itself into an unrecoverable state
22:17 Ryan_Lane I seriously have no clue what to do right now
22:18 Ryan_Lane the process isn't started
22:18 Ryan_Lane upstart thinks it is
22:18 Ryan_Lane there's no pid file
22:18 semiosis Ryan_Lane: please pastie output of 'ps auxw | grep gluster'
22:19 Ryan_Lane root@labstore3:/etc/init# ps auxw | grep gluster
22:19 Ryan_Lane root      1274  0.0  0.0   7624   932 pts/0    R+   22:19   0:00 grep --color=auto gluster
22:19 semiosis heh
22:20 semiosis what if you 'service glusterd stop' then 'service glusterd start'?
22:20 semiosis s/glusterd/glusterfs-server/g
22:20 glusterbot What semiosis meant to say was: what if you 'service glusterfs-server stop' then 'service glusterfs-server start'?
22:20 Ryan_Lane glusterd?
22:20 Ryan_Lane you mean glusterfs-server?
22:20 semiosis yes
22:20 Ryan_Lane stop hangs
22:20 semiosis ooh
22:21 semiosis never seen this before
22:21 Ryan_Lane I've seen this with upstart often
22:22 Ryan_Lane usually a service gets itself into some weird state and upstart thinks it started it
22:23 Ryan_Lane so, the service is failing to stary
22:23 Ryan_Lane *start
22:23 Ryan_Lane according to the logs
22:24 Ryan_Lane [2013-01-29 22:23:01.630832] E [rdma.c:4993:init] 0-rdma.management: Failed to initialize IB Device
22:24 Ryan_Lane ?
22:24 JoeJulian nope
22:24 JoeJulian that's normal
22:24 Ryan_Lane [2013-01-29 22:23:02.472545] E [glusterd-store.c:2080:glus​terd_store_retrieve_volume] 0-: Unknown key: brick-0
22:24 JoeJulian unless, of course, you're using ib.
22:24 JoeJulian normal
22:24 Ryan_Lane [2013-01-29 22:23:02.710245] E [xlator.c:385:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
22:25 JoeJulian not normal
22:26 Ryan_Lane any ideas?
22:26 JoeJulian You have an /etc/glusterfs/glusterd.vol, and it's the default?
22:26 semiosis +1
22:26 Ryan_Lane how would I know if it's default/
22:27 JoeJulian I don't know apt, so <shrug>.
22:27 JoeJulian rpm -q --verify
22:27 JoeJulian hehe
22:27 Ryan_Lane I thought the config files were in /var now?
22:27 Ryan_Lane ah. it specifies that. heh
22:27 JoeJulian The state files are in /var
22:27 Ryan_Lane http://dpaste.com/899041/
22:28 glusterbot Title: dpaste: #899041 (at dpaste.com)
22:28 semiosis https://github.com/gluster/gluste​rfs/blob/master/doc/glusterd.vol
22:28 glusterbot <http://goo.gl/ZetWa> (at github.com)
22:28 purpleidea amccloud: partner: my puppet module goes out of it's way to do *more*, however some sysadmins don't want that. technically it's probably more dangerous, but it's more useful too. it's great if you want to spin up X machines to play with gluster. it literally does 100% of what you need to get going.
22:28 Ryan_Lane looks default to me
22:29 partner purpleidea: but does it replace broken bricks and stuff by itself?
22:29 purpleidea partner: no, for a few reasons (although patches are welcome)
22:29 semiosis wb purpleidea
22:30 purpleidea partner: for one, this is an incredibly hard thing to automate. i'm open to ideas on how to do it. 2) when data is at risk, we want the sysadmin to make an informed decision of what should happen.
22:30 purpleidea semiosis: wb?
22:30 semiosis welcome back :)
22:30 partner purpleidea: i guess that's what we were discussing earlier, how exactly far one wants to take it. nevertheless i'm happy to see such stuff being developed
22:30 purpleidea semiosis: oh, thanks hi. i'm always some where, but i don't always actively follow. always around for pings though.
22:31 Ryan_Lane here's the full paste of the startup errors: http://dpaste.com/899048/
22:31 glusterbot Title: dpaste: #899048 (at dpaste.com)
22:31 partner purpleidea: indeed it is a tricky topic, hence i am at this point mostly thinking of not doing it via chef|puppet|cfengine
22:32 purpleidea partner: it's up to you. have a look at the code, and decide what you want. my code is free software, so feel free to patch it, fork it, etc... semiosis is too i believe.
22:32 semiosis +1
22:32 partner yeah, i have the urls open at my browser but didn't review anything yet
22:33 partner i better get more familiar with the gluster first probably..
22:33 purpleidea partner: what i can recommend, is before you go out and buy a 1000000 of hardware, use my module to spin up a dozen vm's, and you'll 1) get familiar with gluster, and 2) know if the amount it automated scared you. i mean it formats partitions and stuff, which is optional, but some people probably like doing this themselves.
22:34 semiosis Ryan_Lane: you're upgrading from 3.3.0 right?
22:34 Ryan_Lane yes
22:36 partner purpleidea: not using puppet so i would have to rewrite it all. been testing out lots of things with virtual machines and deployed something into our environment already but by far not anything more than a noob on this topic
22:37 partner purpleidea: but definately will read your and semiosis stuff through and see if i want to change my mind and implement something (with our cfengine setup)
22:37 * semiosis scratches his head trying to remember where he's seen "Unknown key: brick-0" before
22:37 purpleidea partner: i'm not a gluster expert either, semiosis and other are far more proficient there, i just wrote a puppet module. good luck!
22:38 JoeJulian semiosis: It's normal.
22:38 semiosis oldbug 3453
22:38 glusterbot Bug http://goo.gl/uknDn is not accessible.
22:38 semiosis figures
22:40 partner purpleidea: sure, that doesn't prevent us from automating the manual steps ;-D
22:40 purpleidea partner: automate all the things
22:41 JoeJulian Ryan_Lane: Can you paste the debug output from "glusterd --debug" please?
22:41 partner purpleidea: nooo, that would make me loose my job since i'm no longer required after that :D
22:41 Ryan_Lane sure. one sec
22:42 purpleidea partner: i've never automated enough that there wasn't anything left to work on... think about it, there's always something to add or improve
22:43 JoeJulian +1
22:43 partner purpleidea: i know, was just joking around there
22:44 Ryan_Lane JoeJulian: http://dpaste.com/899057/
22:44 glusterbot Title: dpaste: #899057 (at dpaste.com)
22:44 purpleidea partner: this gives you more time to submit some patches to gluster...
22:46 partner its a difficult question on how much time one should invest on something you manually do in 1 minute but writing that into a robust trustable promise (cfengine term for recipe) will probably take several days with the testing and stuff..
22:47 purpleidea testing glusterbot: know what a freudian slip is? it's when you say one thing, but mean your mother
22:47 purpleidea s/mother/another/g
22:47 glusterbot What purpleidea meant to say was: testing glusterbot: know what a freudian slip is? it's when you say one thing, but mean your another
22:47 JoeJulian Ryan_Lane: in /var/lib/glusterd/vols/turnkey-mediawiki-project move *.vol somewhere and try glusterd again.
22:47 purpleidea coool
22:47 tomsve joined #gluster
22:48 hattenator joined #gluster
22:49 JoeJulian partner: imo, it's not so much about the 1 min. If you're only doing it every couple of years, it's the 1 hour remembering what you did last time.
22:49 Ryan_Lane JoeJulian: same output
22:50 purpleidea partner: JoeJulian: puppet is the defacto documentation that you now don't need to write in 2013
22:50 purpleidea (because let
22:50 JoeJulian Ryan_Lane: paste /var/lib/glusterd/vols/turn​key-mediawiki-project/info
22:50 purpleidea s face it, i wasn't going to write real docs)
22:50 JoeJulian ^100
22:50 Ryan_Lane JoeJulian: it's an empty file
22:51 JoeJulian Aha!
22:51 JoeJulian Is it not empty on any other server?
22:51 JoeJulian I think you hit a volume sync bug I reported in 3.3.0
22:51 JoeJulian Fixed in 3.3.1 btw
22:51 Ryan_Lane it's not
22:52 partner JoeJulian: yeah, i get your point, its still a matter of putting much effort into automation which you probably still don't trust after half a year as you don't remember what it exactly does, even if its documented there in the configuration management. you'd set up something to test it again from the very beginning
22:52 Ryan_Lane JoeJulian: how do I go about fixing that?
22:52 partner purpleidea: puppet is not defacto for anything but it sure is as good source for the desired end result as any other configuration management system ;)
22:52 Ryan_Lane do I restore bad info files from other servers?
22:53 JoeJulian Ryan_Lane: I can only suggest recreating the volume. rm -rf /var/lib/glusterd/vols/turnkey-mediawiki-project from them all first and recreate it. Without that info file, it's going to fail.
22:53 partner and yes, for a selected topics we do write docs, with the commands used to create this and that, version controlled, shown on server login and so forth
22:53 Ryan_Lane JoeJulian: other servers have info files that have information
22:54 Ryan_Lane only some info files are empty on some systems
22:54 JoeJulian Oh, I thought you were saying it was blank. Just rsync it then.
22:54 Ryan_Lane ok. cool
22:54 Ryan_Lane I was worried when you said recreate the volume :)
22:54 JoeJulian sorry
22:55 Ryan_Lane hm. it's only the two servers I rebooted
22:55 Ryan_Lane that's odd
22:56 JoeJulian It probably tried to sync from the 3.3.0 servers and failed.
22:56 Ryan_Lane ahhhhh
22:56 Ryan_Lane makes sense
23:03 Ryan_Lane JoeJulian: well, I rsync'd them all across
23:03 Ryan_Lane did I need to rsync anything other than the info file?
23:03 Ryan_Lane it's still failing
23:04 Ryan_Lane ah. the bricks directory is empty
23:04 JoeJulian I've never tried anything less than the whole directory before. I'm pretty sure it should regenerate all the .vol files, but I'd just do the whole thing just to make sure.
23:04 Ryan_Lane ah. that's easy enough
23:04 Ryan_Lane ok
23:04 JoeJulian It's easier to have stuff you don't need.
23:05 Ryan_Lane indeed
23:06 Ryan_Lane I should rsync the entire vols directory?
23:06 JoeJulian I have done that.
23:09 Ryan_Lane worked
23:10 JoeJulian yay
23:10 JoeJulian One thing I never understand... why do things that throttle bandwidth (yum, rsync) specify bytes/sec instead of then way we buy it in bits/sec...
23:11 purpleidea JoeJulian: "buy it" ?
23:13 purpleidea JoeJulian: if you mean, why internet companies sell you "megabits/sec" for services, it's because the numbers are bigger, but nobody actually cares about "bits", since nothing is splitting up an actual 8 bit byte.
23:13 JoeJulian Yes, we buy bandwidth in bits/sec. T1, 1.4Mbit/sec, ethernet 1Gbit/sec, oc3 155 Mbit/sec
23:14 andrewbogott joined #gluster
23:14 purpleidea JoeJulian: and i wished we would stop :P i always have to divide by 8 to know how fast these things are in my head.
23:15 JoeJulian So then I have to do math to keep from pissing everyone off when I saturate my (and don't get me started on the lack of available bandwith in this building) 2 T1s upgrading my workstation from Fedora 17.
23:15 purpleidea JoeJulian: it's the same nonsense the HDD companies pulled by using 1000 instead of 1024.
23:15 purpleidea JoeJulian: actually first of all, don't upgrade to F18 yet. I did and there were a bunch of gotchas.
23:15 purpleidea JoeJulian: secondly, get more bandwidth
23:16 JoeJulian I already have at home and on my laptop. There's a few fixes I use.
23:16 * JoeJulian rages.
23:16 purpleidea JoeJulian: Thirdly, well i just use bytes/sec and ignore the bit units unless i read sales docs.
23:16 Ryan_Lane heh. I've been hearing rough things about F18
23:18 purpleidea Ryan_Lane: JoeJulian: FWIW, WIP draft blog post i had of some issues I had and solutions/workarounds: http://ttboj.wordpress.com/?p=34​4&amp;shareadraft=510858983e62c
23:18 glusterbot <http://goo.gl/C4qME> (at ttboj.wordpress.com)
23:18 JoeJulian Plus, it's just my own workstations, not anything that anyone else has to rely on. Worst case, I have a dozen other machines I can use.
23:19 * purpleidea doesn't have nearly as much hardware apparently :P
23:20 JoeJulian huh.. I haven't seen the black background thing.
23:20 purpleidea if i break it, i get to keep both pieces
23:20 purpleidea JoeJulian: if nautilus is managing your desktop, it doesn't work properly.
23:20 purpleidea (but it used to for me in F17)
23:21 partner how come it seems all the desktops get broken at the same time.. grass isn't any greener on the other side i see..
23:21 JoeJulian I've been gritting my teeth and bearing through gnome3.
23:21 Ryan_Lane well. all the processes are back up
23:21 Ryan_Lane but most of the volumes are now showing as down
23:21 JoeJulian rackerhacker retweeted a post showing that it's almost going to be useable again with the next release.
23:21 purpleidea JoeJulian: https://extensions.gnome.org/​extension/67/dash-click-fix/
23:21 glusterbot <http://goo.gl/s5dTQ> (at extensions.gnome.org)
23:22 purpleidea https://extensions.gnome.org/​extension/120/system-monitor/
23:22 glusterbot <http://goo.gl/MGiU1> (at extensions.gnome.org)
23:22 purpleidea lifesavers
23:22 purpleidea lol usable
23:22 JoeJulian Gah, damned spelling.
23:23 purpleidea JoeJulian: oh sorry, i wasn't commenting on spelling, just that it was a funny comment.
23:23 JoeJulian I blame autocorrect (not using something that has it, but it's better than it being my own damned ineptitude).
23:24 JoeJulian Yeah, I really don't like whoever it is that runs the gnome project. They're not getting free beers if I ever have the opportunity.
23:24 melanor9 joined #gluster
23:24 JoeJulian And the fedora board isn't high on the free beer list either.
23:25 JoeJulian They never should have allowed gnome3 through until it reached feature parity with gnome2.
23:25 JoeJulian imho
23:25 purpleidea JoeJulian: i think it will turn around a bit, my only criticism is that i feel fedora got way unstable, and that we're being used as free testers for RHEL, and that they're releasing too early and not often enough with fixes.
23:25 partner may i have the free beer, i'm using debian ;)
23:26 JoeJulian hehe
23:26 purpleidea JoeJulian: agreed. also i would like some beer.
23:26 JoeJulian Come on up.
23:26 partner purpleidea: we need to go and meet up with joe, free beer, nothing can compete with that
23:27 JoeJulian Speaking of which: http://casitconf.org/casitconf13/
23:27 glusterbot Title: Cascadia IT Conference 2013 (at casitconf.org)
23:27 partner you could fix your fedoras while i could enjoy the refreshments
23:28 JoeJulian I'm leading a hands-on workshop for GlusterFS, OpenStack, OpenShift, Logstash, and Puppet.
23:28 purpleidea if anyone wants to sponsor my travel, i will present whatever you need
23:28 semiosis so turns out it was a glusterfs bug not an upstart design flaw /cc JoeJulian Ryan_Lane
23:28 semiosis nice work troubleshooting that JoeJulian
23:28 Ryan_Lane well, it's both
23:28 partner JoeJulian: nice
23:28 semiosis hahaha
23:28 * JoeJulian pulls semiosis out from under the bus.
23:29 Ryan_Lane upstart is still a broken piece of crap for situations like this :)
23:29 semiosis yeah but still
23:29 JoeJulian purpleidea: The presentation list is already accepted.
23:29 Ryan_Lane if a service dies, upstart shouldn't continue to think it's up
23:29 * JoeJulian throws Lennart under the bus.
23:29 JoeJulian ... again.
23:29 purpleidea JoeJulian: that's okay!
23:30 Ryan_Lane lennart didn't write upstart, did he?
23:30 semiosis upstart was scott remnant not lennart
23:30 Ryan_Lane that's systemd, right?
23:30 semiosis yes
23:30 purpleidea Ryan_Lane: yes
23:30 partner i feel sorry for the bus already
23:30 Ryan_Lane heh
23:30 semiosis rofl
23:30 Ryan_Lane so, I'm still having pretty major issues
23:31 purpleidea i'm not a hater, but did you see this: https://en.wikipedia.org/w/index.php?title=Lennart​_Poettering&amp;diff=533966793&amp;oldid=533948775
23:31 glusterbot <http://goo.gl/owZpo> (at en.wikipedia.org)
23:31 Ryan_Lane almost all of my volumes are down
23:31 purpleidea (since you all love lennart, it's funny)
23:31 JoeJulian I think he pushed upstart into fedora before finally admitting it was broken.
23:31 polenta joined #gluster
23:32 Ryan_Lane my logs constantly just spew this: [2013-01-29 23:32:44.612405] I [socket.c:1798:socket_event_handler] 0-transport: disconnecting now
23:32 JoeJulian He's got to be one of the most hated contributers out there. I'd feel sorry for him if he wasn't such an arrogant prick. ;)
23:33 JoeJulian Ryan_Lane: I have a lot of that too... Been ignoring it since everything's been working.
23:33 * Ryan_Lane sighs
23:33 Ryan_Lane upstart is still hanging on stop
23:35 * JoeJulian looks at that bus again...
23:35 semiosis Ryan_Lane: gluster peer status showing any peers rejected?  check all servers
23:36 Ryan_Lane all show as connected
23:36 semiosis well thats good
23:36 semiosis so whats the problem then?
23:36 Ryan_Lane no clue :)
23:37 Ryan_Lane seems stopping and starting one of the broken volumes fixed it
23:37 semiosis great
23:37 purpleidea Ryan_Lane: I vote for firewall/network issue
23:37 Ryan_Lane I have over 300 volumes
23:37 Ryan_Lane purpleidea: no firewall
23:37 Ryan_Lane definitely not a network issue
23:38 purpleidea Ryan_Lane: high packet loss?
23:38 Ryan_Lane nope
23:38 semiosis JoeJulian: kde4 "don't look back" was forced down user's throats waaay before feature parity with kde3, people revolted, now it's awesome... seems the same has been going on lately with gnome
23:38 Ryan_Lane this is all related to the upgrade
23:38 Ryan_Lane heh
23:39 Ryan_Lane when you have this many volumes, the cluster goes insane when you start or stop a service
23:39 Ryan_Lane err
23:39 Ryan_Lane start or stop a volume
23:39 semiosis Ryan_Lane: ok right no idea what the problem is... what's teh symptom?
23:40 Ryan_Lane when I run gluster volume status, most of my volumes have one or more down bricks
23:40 JoeJulian When you start glusterd.... yeah... I see that too. I actually will go so far as to firewall the machine, start glusterd, killall glusterfsd, drop the firewall, then start...force each volume in a controlled manner.
23:40 semiosis Ryan_Lane: go to the servers which host those bricks and restart glusterd?
23:41 semiosis or what JoeJulian said
23:41 Ryan_Lane I hit 20% CPU usage and about 15% of that being waitio when I run: gluster volume start <volume-name.
23:41 Ryan_Lane same with stop
23:41 greylurk joined #gluster
23:41 JoeJulian Lots of open files?
23:41 Ryan_Lane or create, etc.
23:41 Ryan_Lane probably. over 300 volumes
23:42 Ryan_Lane self heal daemon is still broken in this version, it seems :(
23:42 Ryan_Lane dumps core
23:43 JoeJulian Hmm, I haven't had that problem. Are all the servers 3.3.1 now?
23:43 Ryan_Lane well, shit
23:43 Ryan_Lane the memory leak is still there too
23:43 Ryan_Lane yes
23:43 JoeJulian leak?
23:44 Ryan_Lane yeah, a gigantic memory leak. I think it's related to gluster create/start/stop etc on volumes
23:44 Ryan_Lane oh wait. maybe it's gone
23:44 JoeJulian I've got only 30 bricks on my servers, but the memory usage is holding steady.
23:45 Ryan_Lane memory is growing, but that's likely due to the bricks coming back online as I stop/start them
23:45 JoeJulian I did reduce the performance.cache-size to 8mb and my 30 bricks hold steady at 16gb.
23:46 JoeJulian well, just under 16 gb...
23:46 Ryan_Lane I had a leak in 3.3.0 that would consume 150GB of memory in about a week
23:46 JoeJulian Oh, on the client side?
23:46 Ryan_Lane on the server side
23:46 Ryan_Lane the clients also have a leak in 3.3.0
23:47 JoeJulian hmm, never noticed that one. Did noticed the client one.
23:47 purpleidea Ryan_Lane: i've got to go to sleep, but one of these days can you write a blog post or an email about how you manage all those volumes? maybe i could learn your methodology and add some stuff to my puppet code to improve the experience.
23:47 Ryan_Lane I was waiting till 3.3.1 before I reported them
23:47 Ryan_Lane purpleidea: I have a script that manages them. the info it uses is in ldap
23:47 JoeJulian I did talk to avati about the client one in 3.3.0.
23:47 Ryan_Lane we're adding a plugin to openstack that will manage them, rather than using a script
23:48 purpleidea Ryan_Lane: yikes!
23:48 Ryan_Lane basially I create two volumes for every project I have
23:49 glusterbot New news from resolvedglusterbugs: [Bug 764890] Keep code more readable and clean <http://goo.gl/p7bDp>
23:49 Ryan_Lane and we use acls for managing access, based on instances that exist in a project
23:50 purpleidea Ryan_Lane: interesting... it sounds silly because it adds a layer, but you could also do ldap -> puppet (ecn or similar) -> gluster (good idea if you're storing lots of config stuff in ldap, which you use puppet to act on)
23:51 Ryan_Lane puppet is too slow
23:51 Ryan_Lane and it doesn't run often enough
23:51 purpleidea Ryan_Lane: i became more patient
23:51 JoeJulian lo
23:51 JoeJulian lol
23:51 Ryan_Lane the best solution is to have an openstack plugin that manages them
23:51 Ryan_Lane on project creation it'll create the volumes
23:51 purpleidea Ryan_Lane: fair enough
23:52 Ryan_Lane on instance creation/deletion it'll add/remove them from the acls
23:52 JoeJulian Is that plugin on github?
23:52 purpleidea i really haven't played with openstack at all yet
23:52 Ryan_Lane I'm not sure if it's been written yet
23:52 JoeJulian purpleidea: Come to CasITConf.
23:52 Ryan_Lane andrewbogott: had you written a glusterfs plugin for novs?
23:52 Ryan_Lane *nova
23:53 Ryan_Lane JoeJulian: thanks for the help with getting gluster back up, btw
23:53 JoeJulian You're welcome.
23:53 purpleidea JoeJulian: would like to, but i'm not sure i have the funding to go
23:54 andrewbogott Ryan_Lane:  It's a bit stale, but yes:  https://github.com/andrewbogott/no​vawikiplugins/tree/master/sharedfs
23:54 glusterbot <http://goo.gl/WBoIZ> (at github.com)
23:54 andrewbogott Planning to revive it when we upgrade to essex and/or gluster
23:54 Ryan_Lane folsom, you mean?
23:54 andrewbogott um… yes.
23:54 andrewbogott folsom
23:55 Ryan_Lane heh
23:55 * Ryan_Lane nods
23:55 * andrewbogott sings the alphabet song quietly to himself
23:55 JoeJulian hehe
23:57 purpleidea okay, night folks
23:57 andrewbogott Oh, apparently I also meant 'grizzly' when I said 'gluster'.  sheesh.
23:57 JoeJulian Got the letter right.
23:57 andrewbogott true!
23:57 JoeJulian Goodnight purpleidea
23:57 partner muah, feel most confident seeing first two commit messages saying "Added a mostly broken sharedfs plugin" and "Added the 'delete filesystem' command" :D
23:58 Ryan_Lane oh sweet. the self-heal daemon is fixing itself for some nodes too

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary