Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-06-30

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:33 bala joined #gluster
00:45 marcoceppi joined #gluster
00:50 tdasilva_ joined #gluster
01:13 bala joined #gluster
01:34 DV joined #gluster
01:38 harish_ joined #gluster
01:58 gmcwhistler joined #gluster
02:05 glusterbot New news from newglusterbugs: [Bug 1114403] [SNAPSHOT]: On attaching a new node to the cluster while snapshot create was in progress , one of the snapshots failed with "glusterd quorum not met" <https://bugzilla.redhat.co​m/show_bug.cgi?id=1114403>
02:06 sonicrose joined #gluster
02:06 sonicrose hi all, firstly, just wanted to report back on the issue I reported yesterday.  I tracked down that weird '2.1TB bug' to only occur when stat-prefetch is on.  no problems for me with it turned off
02:08 sonicrose second... does anyone know why if I have a 64bit EL6 OS gluster client I can write to a gluster mount with O_DIRECT no problem, but on a 32 bit EL5 gluster client I cannot open a file with O_DIRECT
02:11 fidevo joined #gluster
02:33 ndevos joined #gluster
02:33 ndevos joined #gluster
02:57 gildub joined #gluster
03:09 bharata-rao joined #gluster
03:09 gildub joined #gluster
03:16 RioS2 joined #gluster
03:26 wgao joined #gluster
03:29 gildub joined #gluster
03:36 pranithk joined #gluster
03:55 prasanthp joined #gluster
03:55 itisravi joined #gluster
03:58 DV__ joined #gluster
04:00 ndarshan joined #gluster
04:01 pk1 joined #gluster
04:03 ppai joined #gluster
04:06 shubhendu joined #gluster
04:11 kanagaraj joined #gluster
04:20 jgardeniers joined #gluster
04:21 jgardeniers left #gluster
04:26 nishanth joined #gluster
04:27 harish joined #gluster
04:34 kdhananjay joined #gluster
04:34 nss joined #gluster
04:36 kshlm joined #gluster
04:53 ramteid joined #gluster
04:56 bala joined #gluster
05:02 pranithk left #gluster
05:04 saurabh joined #gluster
05:15 spandit joined #gluster
05:17 hchiramm__ joined #gluster
05:21 Philambdo joined #gluster
05:22 rjoseph joined #gluster
05:23 RameshN joined #gluster
05:24 rastar joined #gluster
05:26 ekuric joined #gluster
05:30 vpshastry joined #gluster
05:32 vimal joined #gluster
05:34 monotek joined #gluster
05:35 hagarth joined #gluster
05:52 rjoseph joined #gluster
05:55 vkoppad joined #gluster
06:00 dusmant joined #gluster
06:04 meghanam joined #gluster
06:04 meghanam_ joined #gluster
06:08 shylesh__ joined #gluster
06:30 Ark joined #gluster
06:32 mortuar joined #gluster
06:33 davinder15 joined #gluster
06:36 hagarth joined #gluster
06:37 lalatenduM joined #gluster
06:46 capri add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 --> Please check that the PPA name or format is correct.
06:53 psharma joined #gluster
06:57 ricky-ti1 joined #gluster
06:59 davinder16 joined #gluster
07:09 bene2 joined #gluster
07:11 notmynicktoo joined #gluster
07:17 ctria joined #gluster
07:21 Alex joined #gluster
07:27 raghu joined #gluster
07:28 raghu2 joined #gluster
07:35 fraggeln good morning dear #gluster, can stat-prefetch be used, or does it still suffers from a memory-leak?
07:40 fsimonce joined #gluster
07:45 dockbram joined #gluster
07:45 dockbram hi all
07:49 fraggeln yello!
07:50 dockbram I've setup gluster 3.5.1 and successfully shared a folder between two nodes. Adding files works fine however deleting a file does not. When I delete a file it stays on one of the nodes. How do I get delete to sync?
07:50 dockbram hi fraggeln
07:56 qdk joined #gluster
07:58 dockbram anyone?
07:59 dockbram So adding a file works instantaneoulsy, removal not
08:00 capri add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.5 --> Please check that the PPA name or format is correct.
08:00 capri what im doing wrong?
08:00 dockbram capri: what's the error?
08:00 dockbram "Please check ..." ?
08:01 capri yes
08:01 dockbram one sec
08:03 dockbram capri: on which distribution is that?
08:03 capri ubuntu
08:03 capri http://download.gluster.org/pub/gluster/​glusterfs/3.5/3.5.1/Ubuntu/Ubuntu.README
08:03 dockbram which version?
08:03 capri 14.04
08:04 dockbram I just tried it on 14.04 LTS and didn't get any error, just the readme
08:04 dockbram can you pastebin or /msg me the exact output?
08:04 bala joined #gluster
08:05 andreask joined #gluster
08:09 bene2 joined #gluster
08:13 dockbram off for lunch, pushed a latest 14.04 LTS/glusterfs 3.5.1 docker image: run with --privileged=true for fuse support bram/glusterfs
08:13 dockbram not configured
08:29 gildub joined #gluster
08:34 ktosiek joined #gluster
08:35 Der_Fisch2 joined #gluster
08:36 capri i tried geo-replication on centos and ubuntu now. but same error on both systems
08:36 capri http://pastebin.com/pMGK2hJg
08:36 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
08:37 capri http://fpaste.org/114290/04117421/
08:37 glusterbot Title: #114290 Fedora Project Pastebin (at fpaste.org)
08:45 dusmant joined #gluster
08:46 prasanthp joined #gluster
08:48 andreask joined #gluster
08:57 saurabh joined #gluster
08:58 vpshastry joined #gluster
09:00 rastar joined #gluster
09:01 mbukatov joined #gluster
09:10 [o__o] joined #gluster
09:25 kshlm joined #gluster
09:26 hagarth joined #gluster
09:32 bala joined #gluster
09:34 shubhendu_ joined #gluster
09:39 romero joined #gluster
09:45 calum_ joined #gluster
09:49 dusmant joined #gluster
09:56 bala joined #gluster
09:57 kumar joined #gluster
09:57 vpshastry joined #gluster
10:00 kshlm joined #gluster
10:17 kanagaraj joined #gluster
10:17 kdhananjay joined #gluster
10:17 n0de joined #gluster
10:22 ctria joined #gluster
10:25 [o__o] joined #gluster
10:26 harish_ joined #gluster
10:27 bala joined #gluster
10:51 DV__ joined #gluster
10:59 DV joined #gluster
11:02 rwheeler joined #gluster
11:06 bene2 joined #gluster
11:18 dusmant joined #gluster
11:20 lalatenduM joined #gluster
11:27 LebedevRI joined #gluster
11:31 sputnik13 joined #gluster
11:34 davinder16 joined #gluster
11:35 qdk joined #gluster
11:41 getup- joined #gluster
11:42 getup- hi, i'm seeing a split brain with volume info split-brain on the path /, not a specific file, how can that happen?
11:44 getup- or how to resolve it for that matter
11:45 vincent_vdk joined #gluster
11:54 deepakcs joined #gluster
12:01 dusmant joined #gluster
12:05 firemanxbr joined #gluster
12:06 edward1 joined #gluster
12:08 diegows joined #gluster
12:10 ndarshan joined #gluster
12:11 [o__o] joined #gluster
12:17 firemanxbr joined #gluster
12:18 itisravi_ joined #gluster
12:19 kshlm joined #gluster
12:28 plarsen joined #gluster
12:29 andreask joined #gluster
12:32 testing123 joined #gluster
12:35 theron joined #gluster
12:36 spandit joined #gluster
12:43 Thilam joined #gluster
12:44 Ark joined #gluster
12:45 B21956 joined #gluster
12:52 julim joined #gluster
12:54 chirino joined #gluster
12:55 julim joined #gluster
12:56 tdasilva joined #gluster
12:59 ghghz joined #gluster
13:00 ghghz Hey, what's the best way to see what's eatting space in gluster? Because runnning "du" it's fatality
13:02 dusmant joined #gluster
13:03 julim joined #gluster
13:11 Thilam|work joined #gluster
13:14 shubhendu_ joined #gluster
13:15 japuzzo joined #gluster
13:17 ndarshan joined #gluster
13:19 nishanth joined #gluster
13:22 glusterbot New news from newglusterbugs: [Bug 1114604] [FEAT] Improve SSL support <https://bugzilla.redhat.co​m/show_bug.cgi?id=1114604>
13:27 n0de joined #gluster
13:29 theron joined #gluster
13:33 ndk joined #gluster
13:38 mjsmith2 joined #gluster
13:43 gmcwhistler joined #gluster
13:45 jcsp joined #gluster
14:09 wushudoin joined #gluster
14:13 fsimonce joined #gluster
14:16 coredump joined #gluster
14:21 theron joined #gluster
14:35 nullck joined #gluster
14:36 kanagaraj joined #gluster
14:37 plarsen joined #gluster
14:40 simulx joined #gluster
14:43 nishanth joined #gluster
14:45 chirino joined #gluster
14:45 diegows joined #gluster
14:52 mortuar joined #gluster
14:52 ghghz left #gluster
15:00 simulx joined #gluster
15:04 simulx joined #gluster
15:05 Guest62746 Anyone know if I will see issues with mounting a glusterfs 3.3 volume with 3.4 client packages?
15:11 harish_ joined #gluster
15:11 andreask joined #gluster
15:20 nueces joined #gluster
15:27 rotbeard joined #gluster
15:28 theron joined #gluster
15:31 diegows joined #gluster
15:38 bennyturns joined #gluster
15:39 bennyturns joined #gluster
15:40 bene2 joined #gluster
15:40 lmickh joined #gluster
15:42 plarsen joined #gluster
15:52 [o__o] joined #gluster
15:53 [o__o] joined #gluster
15:56 theron joined #gluster
15:56 jobewan joined #gluster
15:56 [o__o] joined #gluster
15:57 semiosis Guest62746: that will probably not work
15:58 theron_ joined #gluster
15:59 semiosis dockbram: deletion should be immediate
16:03 Peter1 joined #gluster
16:04 andreask joined #gluster
16:05 theron joined #gluster
16:06 zerick joined #gluster
16:10 saurabh joined #gluster
16:11 SFLimey joined #gluster
16:16 [o__o] joined #gluster
16:23 glusterbot New news from newglusterbugs: [Bug 1114680] [FEAT] Support heterogeneous bricks <https://bugzilla.redhat.co​m/show_bug.cgi?id=1114680>
16:26 ndk joined #gluster
16:29 Peter1 what are these Error messages means from etc-gluster log?
16:29 Peter1 [2014-06-30 15:30:23.197216] E [glusterd-rpc-ops.c:675:__g​lusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: 4a21155b-d64f-4f88-a05f-11fc8346f83f
16:29 Peter1 [2014-06-30 15:30:23.197386] E [glusterd-rpc-ops.c:675:__g​lusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: 731bedcb-f07a-4023-9565-dcae6e81998e
16:29 Peter1 [2014-06-30 15:30:23.197426] E [glusterd-rpc-ops.c:675:__g​lusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: baf862a0-a1bd-40b8-8073-25ffde6d113f
16:29 Peter1 [2014-06-30 15:30:23.197954] E [glusterd-rpc-ops.c:675:__g​lusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: 7c6ac836-9ee1-4d90-9093-364b31db9a3a
16:29 Peter1 and also these
16:29 Peter1 [2014-06-30 15:52:57.237565] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 4a21155b-d64f-4f88-a05f-11fc8346f83f, lock held by: 7c6ac836-9ee1-4d90-9093-364b31db9a3a
16:29 Peter1 [2014-06-30 15:52:57.237575] E [glusterd-handler.c:610:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
16:29 Peter1 [2014-06-30 15:53:29.430583] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 4a21155b-d64f-4f88-a05f-11fc8346f83f, lock held by: 7c6ac836-9ee1-4d90-9093-364b31db9a3a
16:29 Peter1 [2014-06-30 15:53:29.430591] E [glusterd-handler.c:610:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
16:30 JoeJulian Sorry, channel. I'll fix glusterbot to stop that again.
16:31 necrogami joined #gluster
16:31 necrogami joined #gluster
16:33 davinder16 joined #gluster
16:34 JoeJulian @later tell sonicrose The EL5 kernel's fuse does not support O_DIRECT.
16:34 glusterbot JoeJulian: The operation succeeded.
16:35 JoeJulian dockbram: You're adding and deleting files from your volume via a mounted client, yes? Not directly on the bricks?
16:37 JoeJulian Peter1: first of all: ,,(paste)
16:37 glusterbot Peter1: For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
16:37 Peter1 o sorry
16:37 JoeJulian Peter1: second, it means that one of your servers is holding an exclusive lock and won't let go.
16:38 Peter1 is that a bug?
16:38 Peter1 or something we need to worry about?
16:38 JoeJulian What log is that from?
16:38 Peter1 etc-glusterfs-glusterd.vol.log
16:39 cfeller joined #gluster
16:39 Mo__ joined #gluster
16:39 ramteid joined #gluster
16:41 JoeJulian It might be troublesome. It could prevent volume changes and might mean (and this is just hypothetical) that some volume change didn't succeed on one of the servers. restarting glusterd on 7c6a* should clear it (see gluster peer status for the list of uuids). If it restarts fine, I think you're ok. If not, I would guess a volume definition mismatch which can be fixed with rsync.
16:42 Peter1 but i got the log on all nodes
16:43 Peter1 http://ur1.ca/hn2rg
16:43 glusterbot Title: #114394 Fedora Project Pastebin (at ur1.ca)
16:43 JoeJulian Sure, but "lock held by: 7c6ac836-9ee1-4d90-9093-364b31db9a3a".
16:44 Peter1 gluster peer status shows all ok
16:44 JoeJulian yep
16:45 Peter1 the log seems happend quite often and how can i tell if the lock still exist?
16:46 JoeJulian killall -USR1 glusterd and look in the dump file maybe? I'm not sure if that works for glusterd but since its the same binary seems like a good chance.
16:49 Peter1 http://ur1.ca/hn2sa
16:49 glusterbot Title: #114395 Fedora Project Pastebin (at ur1.ca)
16:49 Peter1 this is the full E logs on one nodes
16:49 necrogami joined #gluster
16:49 Peter1 as we can see seems like many locks from all over the nodes?
16:50 pureflex joined #gluster
16:51 Peter1 are there any setting on gluster to tune for locking?
16:51 JoeJulian capri: My guess is that it's having trouble running gsyncd, failing to normalize the url. That's all I can think of after trying to parse that code.
16:51 JoeJulian Peter1: Have you tried my solution yet?
16:52 Peter1 u mean restart one node?
16:52 JoeJulian No, restart glusterd
16:52 JoeJulian Why would I want you to restart a printer? ;)
16:52 Peter1 haha let me try now
16:53 Peter1 how should i just restart glusterd only?
16:53 JoeJulian yes
16:53 JoeJulian oh, how
16:53 JoeJulian which distro?
16:53 Peter1 ubuntu
16:53 JoeJulian service glusterfs-server restart
16:53 Peter1 thanks!
16:54 * JoeJulian dies a little knowing that...
16:54 Peter1 done
16:54 JoeJulian btw, semiosis, I officially hate ubuntu more than ever and consider it to be the least server friendly distro of any known distro.
16:55 Peter1 O
16:55 Peter1 how and what should i verify if everything ok?
16:55 semiosis JoeJulian: http://www.youtube.com/watch?v=pWdd6_ZxX8c
16:55 glusterbot Title: Yeah, well, thats just, like, your opinion, man. - YouTube (at www.youtube.com)
16:55 JoeJulian hehe
16:56 Peter1 peer status and volume status looks ok after restarted glusterd
16:56 JoeJulian I love how, on a headless server, if there's a broken raid initramfs will drop you to a shell regardless of whether or not the raid is a necessary component to the operating system.
16:57 JoeJulian Peter1: Make a junk volume and delete it maybe?
17:00 sputnik13 joined #gluster
17:01 sputnik13 joined #gluster
17:05 realdannys joined #gluster
17:05 Peter1 i am able to create start and stop and delete volume
17:05 Peter1 but then i got these messages on each operations
17:06 realdannys Hi guys, ive got an issue with my gluster data folder. I want it to be owned by nginx:nginx so the computer its mounted on can upload to it from the web browser, however, ive “chown” on both the gluster volume and the mounted server and it always reverts back to root:root ownership - how do I change this permentaly?
17:06 Peter1 http://ur1.ca/hn2um
17:06 glusterbot Title: #114403 Fedora Project Pastebin (at ur1.ca)
17:06 Matthaeus joined #gluster
17:11 Peter1 is that something need to concern?
17:12 JoeJulian Peter1: nope
17:12 semiosis realdannys: you should only need to do that through a client mount.
17:12 Peter1 cool, as those are marked as Error
17:12 Peter1 trying to figure what kind of Error message should be monitor
17:13 realdannys @smiosis well I chown’ed on the mount and it worked temporarily but revents back to root and breaks up loads again
17:13 realdannys I then found a page which said that it mounts according to the original folder permissions, so I went to the original folder, chown’ed that from root:root ro nginx:nginx but after a server reboot the data folder is back to root:root again! (and thus it is on the client as well)
17:14 realdannys @semiosis
17:14 glusterbot realdannys: I do not know about 'semiosis', but I do know about these similar topics: 'semiosis tutorial'
17:14 JoeJulian realdannys: unmount the volume. Chown the directory. mount the volume again.
17:15 realdannys on client or server?
17:15 JoeJulian client
17:15 JoeJulian You need to set the ownership of the directory where the client is *going* to mount.
17:15 semiosis what???
17:15 JoeJulian That's what he just said up there.
17:15 semiosis well thats news to me
17:15 JoeJulian "it mounts according to the original folder permissions"
17:16 semiosis thats not what i've seen
17:16 JoeJulian I'm just basing it on that. I solved that problem years ago and never documented it...
17:16 semiosis in my experience, if you chown the root dir of a volume through a client mount, that goes through to the brick roots.  then future client mounts get it too, regardless of what the mount point on the client was before mount
17:17 JoeJulian I thought I chowned after it was mounted and it stuck forever. If that's what happened, then that would explain why I never documented it because it just seems so obvious.
17:17 semiosis thats how it works for me
17:17 JoeJulian selinux maybe?
17:17 realdannys Well its not getting stuck on mine
17:17 semiosis realdannys: i'd look for something changing it back.  puppet?
17:18 jonathanpoon joined #gluster
17:18 realdannys ive tried chown’ed it on client and the original server
17:18 realdannys when I said “original folder permissions” i meant on the gluster server - someone wrote on serverfault that it reverts to the permissions of the original folder being mounted
17:18 realdannys no puppet or anything complex on the server
17:19 realdannys ive also gone back to an old problem I had of gluster server not starting on boot - someone last time suggested putting a pause in, (as it starts fine manually after boot) as it appears to be trying to start it too soon can anyway remind me how to do that?
17:20 realdannys and finally I have a folder in the data folder .glusterfs which is keeping all the data its had in it - is this supposed to exist? Its filling the volume, I wondered if it was some kind of cache which will automatically prune?
17:21 JoeJulian @lucky what is this new .glusterfs directory
17:21 glusterbot JoeJulian: http://joejulian.name/blog/what-is-​this-new-glusterfs-directory-in-33/
17:22 ghenry joined #gluster
17:23 realdannys ah thanks for that JoeJulian
17:23 realdannys is there anyway to turn that feature off?
17:23 semiosis no
17:23 JoeJulian why would you?
17:23 realdannys ok, i guess i’ll just have to get our script to delete files in both when its finished processing
17:23 realdannys because its wasting hard disk space for me
17:24 realdannys and it means I have to program a script to delete two files
17:24 JoeJulian Not unless you're doing something wrong.
17:24 semiosis the .glusterfs directory contains hard links (for files) and symlinks (for dirs) it should only add a miniscule amount of space (the total length of all the symlink paths)
17:24 JoeJulian don't touch the bricks.
17:24 realdannys oh, well thats not whats happening with me
17:24 realdannys its grown to GBs?!
17:25 JoeJulian Does your script delete files directly off the bricks?
17:26 realdannys Ive got files like this - https://www.dropbox.com/s/4gw759mc5xav16​7/Screenshot%202014-06-30%2018.26.02.jpg
17:26 glusterbot Title: Dropbox - Screenshot 2014-06-30 18.26.02.jpg (at www.dropbox.com)
17:26 semiosis those are hard links, they dont use up space
17:26 realdannys yes, it runs on the AWS which runs gluster, the files are uploaded on the web front end, and all the processing and grunt work is done on our gluster server
17:26 realdannys how is a 100.8mb file not using up space?
17:27 JoeJulian DO NOT DIRECTLY MODIFY THE BRICKS!
17:27 semiosis the actual bytes are stored once, and there are two pointers (hard links) to the bytes in the directory tree
17:27 semiosis when all hard links are removed the bytes are freed
17:27 realdannys they are all deleted
17:27 JoeJulian that's like editing the inode tables to muck with file entries on a mounted filesystem.
17:28 realdannys semiosis, they are deleted once processing is finished
17:28 * JoeJulian throws up his hands.
17:28 semiosis ok so JoeJulian is like 10 steps ahead of me.  you should listen to him
17:28 realdannys so are you saying JoeJulian we have to move the zip files from the brick to another place on the server?
17:28 realdannys isn’t that modifying them too?
17:28 semiosis realdannys: see ,,(glossary)
17:28 glusterbot realdannys: A "server" hosts "bricks" (ie. server1:/foo) which belong to a "volume"  which is accessed from a "client"  . The "master" geosynchronizes a "volume" to a "slave" (ie. remote1:/data/foo).
17:29 semiosis a brick is a specific directory dedicated to be backend storage for a glusterfs volume.
17:29 semiosis once you make a brick you should only modify it through a glusterfs client mount point
17:29 semiosis if you need to delete a file, make a local client mount & delete through there
17:30 semiosis afk
17:30 realdannys right ok
17:30 realdannys I wasn’t aware of this
17:30 realdannys so I need to setup a local client amount and have our script run on that
17:30 realdannys *mount
17:32 realdannys Ok, so i’ll figure out how to do a local client mount now and move all the processing over to that.
17:32 realdannys is it ok to delete the .glusterfs folder though yes? I have no files on the mount at the minute and thus no need for that folder either
17:32 JoeJulian yes, you can do that. Then you must run a "gluster volume heal $vol full"
17:33 realdannys ok, cool, i’ll do that
17:33 realdannys and finally, how can I make the gluster server pause on boot, someone told me how to do it and ive forgotten as we thought it was fixed after a reboot but on a cold boot it never starts
17:34 daMaestro joined #gluster
17:37 realdannys anyone?
17:39 JoeJulian Throw a sleep in mount.glusterfs maybe?
17:40 cmtime someone buy JoeJulian a beer he deserves probably a million of them.
17:40 JoeJulian If I ever have a chance to take a world tour, I won't remember it for all the beers owed.
17:40 cmtime Ya
17:43 cmtime Hey JoeJulian do you know of any bugs in quota?  I have 4 different glusters and on all of them when I turn quota it locks the gluster up. In the end I have to disable quota to be able to access files again.  In my test setup I have no problems but in production with several billion files I have problems.
17:44 cmtime And if you ever are in Montreal and need some free Beers I can make sure some people take you out.
17:47 JoeJulian Hey, you and purpleid1a should meet up.
17:48 cmtime hehe I am not in Montreal but my company is and I could make sure they show you a good time.
17:49 realdannys JoeJulian - that was the one, how did I did that exactly?
17:50 realdannys although remember its not the client, its the server, “glusterd” is not starting on boot
17:50 realdannys but its fine if I go and start it manually, it just attempts to load too early
17:51 JoeJulian Oh, that one... I did that in the upstart job...
17:52 realdannys no idea why, occasioanlly I reboot, and it’ll load the service just fine, other times, especially from a cold boot, it wont start unless I SSH in and manually start it
17:52 realdannys you’d think 1’s and 0’s would behave the same every time
17:53 JoeJulian Probably has something to do with your network not starting up in the same amount of time.
17:53 realdannys yes I guess - im running on EC2 instances
17:53 realdannys so where could I put the sleep for the glusterd service?
17:56 chirino joined #gluster
17:57 realdannys WTF..? Now when I mount the gluster on teh client its mounting as mysql:memcached
17:57 realdannys ??!!
17:57 realdannys all the folders were set to nginx:nginx
17:58 JoeJulian uid's not synced?
17:58 realdannys meaning?
17:59 realdannys I can chown and change it
17:59 realdannys lets see what happens if I unmount now
17:59 realdannys so now its
17:59 realdannys https://www.dropbox.com/s/7pekm7n9bsljav​o/Screenshot%202014-06-30%2018.59.21.jpg
17:59 glusterbot Title: Dropbox - Screenshot 2014-06-30 18.59.21.jpg (at www.dropbox.com)
17:59 JoeJulian grep nginx /etc/passwd
18:00 realdannys ive unmounted and its - https://www.dropbox.com/s/gw9unktc8zwbgj​v/Screenshot%202014-06-30%2018.59.52.jpg
18:00 glusterbot Title: Dropbox - Screenshot 2014-06-30 18.59.52.jpg (at www.dropbox.com)
18:00 realdannys ok its stuck now on remount
18:00 realdannys and that came back as “nginx:x:496:494:nginx user:/home/nginx:/sbin/nologin"
18:01 JoeJulian Now do that on the other machine.
18:01 realdannys the other maschine (the gluster server) is
18:01 realdannys nginx:x:495:495:nginx user:/var/cache/nginx:/sbin/nologin
18:02 JoeJulian Notice how one says 495 and the other 496? Those are the uid
18:02 realdannys ok, and they should be the same?
18:02 JoeJulian @lucky uid gid
18:02 glusterbot JoeJulian: https://kb.iu.edu/d/adwf
18:02 JoeJulian Meh, not what I was hoping.
18:03 realdannys if they both need to be the same, how can I go about changing that?
18:03 JoeJulian I ensure it with puppet.
18:05 JoeJulian I don't know in ubuntu. It's a complicated issue because all the files that are supposed to be owned by nginx are assigned that uid/gid. If another package too the other uid/gid on that machine then you'd have to fix all of that one too.
18:06 JoeJulian s/package too/package took/
18:06 glusterbot What JoeJulian meant to say was: I don't know in ubuntu. It's a complicated issue because all the files that are supposed to be owned by nginx are assigned that uid/gid. If another package took the other uid/gid on that machine then you'd have to fix all of that one too.
18:06 JoeJulian In centos/rhel/fedora they coordinate service uid/gid so that doesn't happen.
18:07 JoeJulian I would hope that if the user and group (nginx) was created before installing the packages, it would use the pre-existing one.
18:08 realdannys im on Centos6
18:09 JoeJulian Oh! then wtf? :)
18:11 _Bryan_ joined #gluster
18:12 realdannys well, i could try rebooting now ive mounted and unmoutned a few times with ownership staying in tact
18:13 realdannys but could you give a pointer to make the glusterd service delay on boot so I can test that too?
18:17 semiosis i use the range 200-300 for uid/gid important to my app
18:17 semiosis with puppet, i just ensure that the user exists before the package gets installed
18:20 JoeJulian realdannys: Since you're using centos, add a sleep to /etc/init.d/glusterd in the start() function.
18:22 realdannys ok line 52 ive got
18:22 realdannys # Start the service $BASE
18:22 realdannys start()
18:22 realdannys you’ll have to hold my hand here, how do I enter sleep to the start ()
18:22 realdannys never done it before, total newbir
18:22 realdannys newbie
18:23 semiosis why are you delaying glusterd?
18:23 realdannys its not starting on boot reliably, it tries to start too early
18:24 semiosis why isnt it starting ok?
18:24 semiosis do you get an error in the log?
18:25 dtrainor joined #gluster
19:03 realdannys joined #gluster
19:03 realdannys Bah computer crashed and then I was stuck in updates!
19:04 * JoeJulian flips the daily bird toward Redmond.
19:04 mortuar joined #gluster
19:05 realdannys erm, JoeJulian, if you answered the sleep question for me, could you copy and paste the answer again for me?
19:06 semiosis there's a link to the channel logs in the topic just type
19:06 semiosis /topic
19:06 JoeJulian semiosis had a valid question, though. Why isn't it starting ok? Do you get an error in the log?
19:10 realdannys ermm, we went through it last time, I believe it just tries to start before the network is up
19:10 realdannys as you suggested
19:11 Ark joined #gluster
19:12 JoeJulian What *I* would do, rather than mess with a packaged init file... I would create my own init file that does nothing more than sleep and put its start order between network and glusterd
19:12 realdannys heres my boot log - http://hastebin.com/jewoqecoba.sm
19:12 glusterbot Title: hastebin (at hastebin.com)
19:12 realdannys just says failed for starting glusterd
19:13 JoeJulian The log is /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
19:13 JoeJulian well, it says the network is up.
19:14 realdannys and here is the gluster log
19:14 realdannys http://hastebin.com/ijuhicefew.sm
19:14 glusterbot Title: hastebin (at hastebin.com)
19:14 realdannys the first is the failed attempt at boot, the second succesful is my manual start at the command line
19:20 JoeJulian So you're using hostnames like ec2-23-21-168-59.compute-1.amazonaws.com ? What if the IP changes?
19:20 realdannys it doesn’t they’re elastic IP’s
19:21 realdannys always the same
19:21 JoeJulian Ah, ok
19:21 realdannys maybe, just maybe, its because EC2 hasn’t mounted the EBS volume in time for gluster
19:21 realdannys that could very well be the answer - and would especially explain why its more prevelent on cold boots
19:21 realdannys being as the EBS is effectively an external network drive to the server
19:22 JoeJulian Is /var/lib/gluster on ebs?
19:22 realdannys I think the entire drive for the server is an ebs volume
19:22 JoeJulian That would be my guess, in which case it wouldn't have gotten that far without it.
19:23 realdannys actually there are two drives
19:23 realdannys and the data folder is on the 20gb EBS I believe - although thinking back ive no idea how I arranged that
19:23 JoeJulian I would assume one of them is the brick.
19:23 realdannys but I would have a guess that its the brick which isn’t online in time occasioanlly
19:24 realdannys so yeah, causing the server not to start
19:24 JoeJulian let's see fstab
19:24 realdannys so a pause would be the best way to create a safety net around this
19:24 realdannys http://hastebin.com/yipapigiyu.hs
19:24 glusterbot Title: hastebin (at hastebin.com)
19:24 realdannys fstab
19:24 JoeJulian No, the "best" way uses tests and logic. :P
19:25 realdannys yes /dev/xvdf is the 20gb ebs volume, so its not coming up in time for gluster on cold boots
19:26 JoeJulian That's mounted during "local" mounts, so it's up before the network starts.
19:26 realdannys hmmm ok
19:26 realdannys well I have no idea - but for some reason waiting and running it from teh command line works perfectly :/
19:28 fraggeln mr JoeJulian: Hello
19:28 JoeJulian realdannys: edit /etc/sysconfig/glusterd and add GLUSTERD_LOGLEVEL="DEBUG" and produce the failure again.
19:28 JoeJulian brb
19:28 realdannys ok
19:33 julim joined #gluster
19:33 mjsmith2 joined #gluster
19:40 Gugge joined #gluster
19:54 glusterbot New news from newglusterbugs: [Bug 1058300] VMs do not resume after paused state and storage connection to a gluster domain (they will also fail to be manually resumed) <https://bugzilla.redhat.co​m/show_bug.cgi?id=1058300>
19:54 realdannys JoeJulian - here it is after reboot, I dont see any differece in the log level though? http://hastebin.com/alojuzaqus.sm
19:54 glusterbot Title: hastebin (at hastebin.com)
19:58 gmcwhistler joined #gluster
20:00 mjsmith2 joined #gluster
20:02 julim joined #gluster
20:13 gomikemike joined #gluster
20:18 realdannys joined #gluster
20:33 andreask joined #gluster
20:35 realdannys You back JoeJulian?
21:02 andreask joined #gluster
21:02 andreask joined #gluster
21:07 mjsmith2 joined #gluster
21:28 radez_g0` joined #gluster
21:29 abyss___ joined #gluster
21:29 rwheeler_ joined #gluster
21:30 Rydekull_ joined #gluster
21:30 jezier joined #gluster
21:30 saltsa_ joined #gluster
21:32 FooBar_ joined #gluster
21:34 atrius joined #gluster
21:35 nhayashi_ joined #gluster
21:36 Alex joined #gluster
21:42 Slasheri joined #gluster
21:42 Slasheri joined #gluster
21:42 daMaestro joined #gluster
21:45 zerick joined #gluster
21:46 redbeard joined #gluster
21:54 SFLimey joined #gluster
22:03 n0de joined #gluster
22:08 sage_ joined #gluster
22:34 Peter1 JoeJulian: I still keep getting the "unable to get lock" Error on etc log
22:34 Peter1 http://fpaste.org/114480/14041676/
22:34 glusterbot Title: #114480 Fedora Project Pastebin (at fpaste.org)
22:35 Peter1 wonder what could cause these locking error
22:36 JoeJulian I don't know... I'll try to ask a dev more about it during Bangalor's waking hours.
22:36 Peter1 thanks!
22:36 Peter1 also how about these warnings?
22:36 Peter1 http://fpaste.org/114481/14041677/
22:36 glusterbot Title: #114481 Fedora Project Pastebin (at fpaste.org)
22:36 Peter1 i am only using NFS for now
22:39 JoeJulian Not strong on NFS, but I would guess that has something to do with anonymous fd.
22:41 realdannys jJoeJulian - here it is after reboot, I dont see any differece in the log level though? http://hastebin.com/alojuzaqus.sm
22:41 glusterbot Title: hastebin (at hastebin.com)
22:41 JoeJulian Me neither.
22:44 JoeJulian Well that's why... the example is wrong. It's actually just supposed to be LOG_LEVEL not GLUSTERD_LOGLEVEL
22:53 plarsen joined #gluster
22:58 realdannys oh, doh
23:00 realdannys So changed #GLUSTERD_LOGLEVEL="DEBUG" to #LOGLEVEL="DEBUG"
23:00 JoeJulian Do you know anything about bash scripting?
23:01 realdannys very little, so no, not really
23:01 JoeJulian (Not a put down, just looking for context)
23:01 JoeJulian Ok, comments in bash start with "#". So that line does absolutely nothing.
23:02 ThatGraemeGuy joined #gluster
23:03 realdannys ah yes, I did notice that, ive deleted the # now
23:04 realdannys stopping and starting again
23:08 mjsmith2 joined #gluster
23:09 realdannys still nothing new in the log :/
23:19 mjsmith2 joined #gluster
23:24 RioS2 joined #gluster
23:35 andreask joined #gluster
23:50 realdannys finally
23:50 realdannys JoeJulian: Here is the debug log http://hastebin.com/doyahizeni.sm
23:50 glusterbot Title: hastebin (at hastebin.com)
23:51 JoeJulian So is ec2-23-21-168-59.compute-1.amazonaws.com itself, or it's peer?
23:52 * JoeJulian grubles that the Error should have included the hostname that failed to resolve.
23:53 * JoeJulian also grumbles
23:56 realdannys thats itself
23:57 JoeJulian Then the simple solution would be to add that to the 127.0.0.1 /etc/hosts entry.
23:57 realdannys would it? ok
23:59 realdannys ok ive done that, and i’m shutting down again
23:59 realdannys you think it’ll come up ok this tiem?

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary