Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2013-12-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:16 _polto_ joined #gluster
00:16 _polto_ joined #gluster
00:30 cfeller joined #gluster
00:50 glusterbot New news from newglusterbugs: [Bug 1038348] RPM package name <http://goo.gl/2EP3qM>
00:53 Technicool joined #gluster
00:58 davidbierce joined #gluster
01:22 cogsu joined #gluster
01:26 itisravi joined #gluster
01:31 lpabon joined #gluster
01:34 jag3773 joined #gluster
01:47 psiphi75 Hi, is there anyway to simulate split brain in Gluster 3.2?  I want to do some testing to ensure this works and that we can heal the system.
01:48 Alex iptables might be easiest? Just firewall off the boxes from eachother
01:55 psiphi75 Thanks, iptables did the trick.  Now the client hangs when it reads the file.  Is this expected behaviour?
02:19 harish joined #gluster
02:44 mistich joined #gluster
02:46 mistich I have a question about bricks is it better to create one brick per server or take each drive and make a brick
02:47 samppah both ways has advantages
02:48 samppah i'm using raid on most of my setups
02:49 samppah if disk dies then it does 't have to transfer everything from another server
02:49 mistich that is what I currently am running but wasn't sure if I would get more performance if I removed the raid
02:49 mistich have a 10gig network between them
02:49 hchiramm_ joined #gluster
02:50 samppah glusterfs writes one file per brick, so one file performance would be then limited to one hdd
02:51 mistich and running a Distributed-Replicate
02:53 mistich so which would I get better performance with multiple bricks per server or one brick per server
02:54 samppah what is your use case?
02:55 mistich small file rrd's
02:55 itisravi joined #gluster
02:57 mistich per the Performance in a Gluster System section 3.1.5 seems like the test were multiple bricks
03:00 samppah ok
03:00 samppah is one file written many times or just once?
03:01 mistich one file ever 30 mins
03:02 mistich but 3 million files :)
03:03 mistich writes are staggered
03:04 mistich current setup is writing 2.2 million no problems
03:04 mistich would like to get more
03:04 mistich thats with ssd drives in a raid 10
03:05 mistich 1 brick per server with 8 servers
03:07 mistich question is if I did 6 bricks per server with 8 server would I get better performance
03:08 samppah ahh.. i had one setup with ssd for vm images, it worked fine in one drive per brick
03:08 samppah with regular hdd i'd probably just stick with raid
03:09 samppah i'm zorry i have to leave now.. got stuff to do in datacenter
03:09 mistich ok thanks
03:13 kshlm joined #gluster
03:13 Alex psiphi75: I'm afraid I can't give you a good answer, sorry!
03:16 semiosis file a bug
03:16 glusterbot http://goo.gl/p7D4Fh
03:31 mistich anyone else which is better 1 brick per server with 8 servers or 6 bricks per server with 8 server which would get better performance
03:37 sgowda joined #gluster
03:37 shubhendu joined #gluster
03:40 rjoseph joined #gluster
03:42 davinder joined #gluster
03:46 diegows joined #gluster
03:50 itisravi joined #gluster
03:50 bharata-rao joined #gluster
03:51 glusterbot New news from newglusterbugs: [Bug 1038391] gf-error-codes.h is missing from source tarball <http://goo.gl/JL19Zg>
03:51 itisravi_ joined #gluster
03:57 mohankumar joined #gluster
03:59 kanagaraj joined #gluster
04:01 semiosis mistich: enough bricks per server to saturate the server's NIC
04:01 semiosis i imagine with ssd that would not take too many bricks
04:07 mistich semiosis so if I break it out more I should get better performance
04:08 semiosis idk for sure
04:08 semiosis generally speaking more servers will be faster than less servers, but every setup is different
04:08 mistich only using 700mb of network have 10 gig cards
04:09 mistich would you know how or a doc to move data off 4 server and add them back then remove 4 and then add them back
04:10 mistich so I do not lose data
04:10 ndarshan joined #gluster
04:11 semiosis rsync?
04:11 semiosis not sure i understand
04:15 mistich no I have 8 bricks want to remove 4 and remove the raid add them back then remove the other 4 and remove the raid without losing data
04:15 saurabh joined #gluster
04:16 mistich I should be able to shrink to 4 by gluster volume remove-brick VOLNAME BRICK
04:16 mistich that would move the data to only 4 bricks
04:16 mistich then do a  gluster volume rebalance VOLNAME start
04:17 mistich to balance out from 8 to 4 is this correct
04:19 semiosis do you have replication on this volume?
04:20 semiosis not sure about this, but i think when you remove-brick the data migrated off to other bricks in a balanced way
04:20 semiosis making a big assumption here though, dont take my word for it
04:20 mistich ok I'll google somemore thanks
04:21 mistich at 2.2 million rrds need to get to 3 million
04:21 mistich almost there :)
04:21 semiosis if you have replication, and want to just replace the underlying storage for a brick (not moving it to another server) you can just take the brick offline, do local operations, then bring the brick back online & self heal the difference
04:21 semiosis that i have done several times without major problems
04:22 mistich Distributed-Replicate
04:22 semiosis personally, i avoid add-brick/remove-brick and rebalance
04:22 _BryanHm_ joined #gluster
04:23 mistich well I will have to destroy the brick to remove the raid 10 then will add back 6 bricks instead of 1 will that be ok
04:24 mistich I'll give it a try
04:24 mistich thanks
04:26 hchiramm_ joined #gluster
04:29 semiosis good luck
04:34 MiteshShah joined #gluster
04:35 raghu joined #gluster
04:36 dylan_ joined #gluster
04:36 ppai joined #gluster
04:38 shyam joined #gluster
04:55 sticky_afk joined #gluster
04:55 nshaikh joined #gluster
04:55 stickyboy joined #gluster
04:56 dusmant joined #gluster
04:57 shylesh joined #gluster
04:58 hagarth joined #gluster
04:59 _pol joined #gluster
05:01 bala joined #gluster
05:02 _pol joined #gluster
05:08 satheesh joined #gluster
05:12 Technicool joined #gluster
05:14 CheRi joined #gluster
05:19 hchiramm_ joined #gluster
05:22 _pol joined #gluster
05:29 sgowda joined #gluster
05:30 bulde joined #gluster
05:30 Eco_ joined #gluster
05:34 psharma joined #gluster
05:37 spandit joined #gluster
05:39 sticky_afk joined #gluster
05:39 stickyboy joined #gluster
05:42 aravindavk joined #gluster
05:45 davinder joined #gluster
05:48 Eco_ joined #gluster
05:48 badone joined #gluster
05:50 ababu joined #gluster
05:54 shubhendu joined #gluster
05:57 ndarshan joined #gluster
05:57 dusmant joined #gluster
05:58 shruti joined #gluster
05:58 bala joined #gluster
05:58 sgowda joined #gluster
06:00 vpshastry1 joined #gluster
06:01 kanagaraj joined #gluster
06:05 hchiramm_ joined #gluster
06:05 rastar joined #gluster
06:10 hagarth joined #gluster
06:19 lalatenduM joined #gluster
06:22 spandit joined #gluster
06:25 anands joined #gluster
06:28 davinder joined #gluster
06:29 krypto joined #gluster
06:45 RameshN joined #gluster
06:48 mistich how do you tell on a Distributed-Replicate when you add a node back that it has replicated all the data back?
06:50 foster joined #gluster
06:51 kanagaraj joined #gluster
06:51 glusterbot New news from newglusterbugs: [Bug 1038442] chown() does not clear suid bit <http://goo.gl/YZT0lN>
06:52 ndarshan joined #gluster
06:52 bala joined #gluster
06:53 aravindavk joined #gluster
06:53 hagarth joined #gluster
06:54 dusmant joined #gluster
07:22 morse joined #gluster
07:23 jtux joined #gluster
07:23 samppah mistich: what version of glusterfs you are using?
07:24 samppah with 3.3 and 3.4 you should see files that need replication with gluster volume heal volName info
07:33 XpineX joined #gluster
07:44 ngoswami joined #gluster
07:47 psyl0n joined #gluster
07:51 hagarth joined #gluster
07:56 shubhendu joined #gluster
07:57 ctria joined #gluster
07:59 dylan_ joined #gluster
08:01 eseyman joined #gluster
08:05 satheesh joined #gluster
08:10 getup- joined #gluster
08:11 keytab joined #gluster
08:15 keytab joined #gluster
08:19 _polto_ joined #gluster
08:19 _polto_ joined #gluster
08:19 shubhendu joined #gluster
08:20 davidbierce joined #gluster
08:24 shyam joined #gluster
08:27 vshankar joined #gluster
08:37 andreask joined #gluster
08:39 StarBeast joined #gluster
08:41 vpshastry joined #gluster
08:42 mbukatov joined #gluster
08:52 samppah hagarth: hmm.. is there any other documentation available about File Snapshot than mentioned here http://www.gluster.org/community/documentation/index.php/Features/File_Snapshot?
08:52 glusterbot <http://goo.gl/4o7aAO> (at www.gluster.org)
08:52 samppah http://review.gluster.org/#/c/5367/ actually here :)
08:52 glusterbot Title: Gerrit Code Review (at review.gluster.org)
08:58 sheldonh mistich: i'd wait for the rebalance to complete
09:00 vimal joined #gluster
09:06 ndarshan joined #gluster
09:11 pkoro joined #gluster
09:15 calum_ joined #gluster
09:20 andreask joined #gluster
09:20 ndarshan joined #gluster
09:20 geewiz joined #gluster
09:21 verywiseman joined #gluster
09:24 hchiramm_ joined #gluster
09:28 meghanam joined #gluster
09:30 abyss^ Can I migrate master gluster (the first gluster) by gluster volume replace-brick?
09:30 hagarth samppah: is that documentation good enough?
09:36 d-fence joined #gluster
09:36 satheesh joined #gluster
09:37 sheldonh_ joined #gluster
09:41 rjoseph joined #gluster
09:44 _pol joined #gluster
09:48 sheldonh_ left #gluster
09:52 vpshastry joined #gluster
09:53 ndarshan joined #gluster
09:54 saurabh joined #gluster
09:59 vpshastry left #gluster
10:00 tycho joined #gluster
10:09 rjoseph joined #gluster
10:12 vpshastry joined #gluster
10:21 davidbierce joined #gluster
10:28 samppah hagarth: i hope i have time to test that out tomorrow.. i'm just bit confused if it's possible to use that with existing vm images (qcow2)
10:28 shyam joined #gluster
10:28 TrDS joined #gluster
10:29 diegows joined #gluster
10:31 samppah also is external snapshot already supported? mentioned as next patch in gerrit
10:31 ppai joined #gluster
10:34 TrDS hi... i'd like to move an existing gluster (3.3.1) into a different subnet with new dns names, but could not find any information on how to rename/move peers (and bricks)... if there is no command for this, is it possible to do it manually, like renaming everything in /var/lib/glusterd, hoping to not forget anything? ;)
10:55 RameshN joined #gluster
10:58 harish joined #gluster
11:03 khushildep joined #gluster
11:06 vpshastry joined #gluster
11:09 ndarshan joined #gluster
11:11 ppai joined #gluster
11:14 dylan_ joined #gluster
11:14 social hagarth: I threw in some updates to https://bugzilla.redhat.com/show_bug.cgi?id=1032122
11:14 glusterbot <http://goo.gl/HbDvQ6> (at bugzilla.redhat.com)
11:14 glusterbot Bug 1032122: unspecified, unspecified, ---, kparthas, NEW , glusterd getting oomkilled
11:22 dusmant joined #gluster
11:22 harish joined #gluster
11:22 Peanut I see glusterbot is still pushing homeimprovement.com :-)
11:24 samppah homeimprovement.com? :)
11:24 samppah oh
11:26 abyss^ someone can help? I have couple of question about migrate bricks on gluster;)
11:27 social abyss^: it's probably better to just ask maybe we'll know
11:29 abyss^ social: yes, thx, but I asked above and nobody answered:)
11:29 abyss^ OK. So: Can I migrate first server of gluster? I need to move all gluster server to another localization.
11:30 social do you have replica? what do you mean by migration?
11:31 abyss^ social: gluster volume replace-brick saas gluster1:/ydp/shared gluster1-saas:/ydp/shared start. I mean smth like this:) But any other working approach is fine for me:)
11:32 andreask joined #gluster
11:33 abyss^ Can I do smth like this: http://pastebin.ca/2492847
11:33 glusterbot Title: pastebin - Someone - post number 2492847 (at pastebin.ca)
11:33 lanning joined #gluster
11:33 abyss^ I mean: I do first replace-brick gluster1:/ydp/shared gluster1-saas:/ydp/shared, then: gluster1:/ydp/shared2 gluster1-saas:/ydp/shared. It's possible?
11:34 andreask joined #gluster
11:34 abyss^ of course in want will change: Number of Bricks to = 2 (if it possible)
11:36 hagarth social: cool, can you add +1 verified for the patch in review.gluster.org?
11:39 Staples84 joined #gluster
11:39 getup- joined #gluster
11:41 shishir joined #gluster
11:43 dusmant joined #gluster
11:44 _pol joined #gluster
11:56 ira joined #gluster
12:02 vpshastry joined #gluster
12:02 shishir joined #gluster
12:03 kkeithley1 joined #gluster
12:11 CheRi joined #gluster
12:13 social abyss^: I think replace brick is fairly safe
12:13 social abyss^: ain it should work
12:16 social hagarth: if I find how
12:16 itisravi joined #gluster
12:18 ppai joined #gluster
12:22 davidbierce joined #gluster
12:23 shishir joined #gluster
12:24 andreask joined #gluster
12:30 ninkotech joined #gluster
12:31 edward2 joined #gluster
12:31 edward2 joined #gluster
12:33 social hagarth: how does one send patches to gluster? any link on howto?
12:39 vpshastry social: http://www.gluster.org/community/documentation/index.php/Development_Work_Flow
12:39 glusterbot <http://goo.gl/ooDX6q> (at www.gluster.org)
12:43 kkeithley_ ,,(patch)
12:43 glusterbot I do not know about 'patch', but I do know about these similar topics: 'paste'
12:43 social from what I understand now I just need to push when everything is ready? :)
12:43 * social never worked with gerrit
12:44 kkeithley_ you run the ./rfc.sh script which will do the push and submit to gerrit
12:44 kkeithley_ @learn submit_patch as "To submit a patch to gluster's gerrit see http://www.gluster.org/community/documentation/index.php/Development_Work_Flow"
12:44 glusterbot kkeithley_: The operation succeeded.
12:45 social hmm fatal: Could not read from remote repository.
12:46 kkeithley_ Did you create an account and upload your ssh pubkey?
12:46 social yep
12:47 satheesh1 joined #gluster
12:51 brimstone left #gluster
12:53 social kkeithley_: I had wrong remote
12:55 hagarth1 joined #gluster
12:57 davidbierce joined #gluster
12:59 psyl0n joined #gluster
13:15 TrDS regarding peer rename/move, in the mailinglist someone else also wrote about messing with /var/lib/glusterd content (http://www.gluster.org/pipermail/gluster-users/2013-September/037313.html)... is this really the only way?
13:15 glusterbot <http://goo.gl/ZO00Dh> (at www.gluster.org)
13:19 Dave2 joined #gluster
13:21 tqrst what do the gluster devs use to spin up test clusters when working on the gluster source? I just came across http://www.gluster.org/2013/10/instant-ephemeral-gluster-clusters-with-vagrant/ but I'm curious if there's anything else I should be looking at.
13:21 glusterbot <http://goo.gl/cN3rzQ> (at www.gluster.org)
13:23 diegows joined #gluster
13:25 davidjpeacock vagrant certainly looks nice
13:27 FooBar I use ansible, but that's only after base-install/virtual-machine creation
13:27 FooBar though it could be used to provision VM's as well
13:28 mattapp__ joined #gluster
13:29 tqrst right
13:30 tqrst (I can also imagine the more dedicated devs probably just have their own test cluster)
13:31 dusmant joined #gluster
13:31 davidjpeacock tqrst: FAI may be of interest too - http://fai-project.org
13:31 glusterbot Title: FAI - Fully Automatic Installation (at fai-project.org)
13:31 CheRi joined #gluster
13:33 tqrst davidjpeacock: interesting
13:33 tqrst looks like a more generic cobbler
13:35 lalatenduM joined #gluster
13:38 onny1 joined #gluster
13:45 _pol joined #gluster
13:47 _amir joined #gluster
13:48 mkzero i just got a lot of 'failed to open stream: Read-only file system' on some of my clients. any idea where this came from?
13:51 JonathanD joined #gluster
13:51 piffio joined #gluster
13:52 bgpepi joined #gluster
13:59 NigelSUK joined #gluster
13:59 NigelSUK Afternoon :)
14:00 davidbierce joined #gluster
14:01 mattapp__ joined #gluster
14:04 mattappe_ joined #gluster
14:12 nhm joined #gluster
14:18 vpshastry left #gluster
14:22 B21956 joined #gluster
14:26 japuzzo joined #gluster
14:29 ababu joined #gluster
14:37 khushildep joined #gluster
14:41 _polto_ joined #gluster
14:41 _polto_ joined #gluster
14:50 bala joined #gluster
14:51 CheRi joined #gluster
14:53 rwheeler joined #gluster
14:57 social how does one setup georeplication in 3.5?  are there any docs to it?
14:59 getup- joined #gluster
15:01 dbruhn joined #gluster
15:03 kaptk2 joined #gluster
15:04 bennyturns joined #gluster
15:04 MrNaviPacho joined #gluster
15:04 lalatenduM joined #gluster
15:05 failshell joined #gluster
15:07 dylan_ joined #gluster
15:14 jbrooks joined #gluster
15:16 rwheeler joined #gluster
15:16 bugs_ joined #gluster
15:18 bala1 joined #gluster
15:20 lpabon joined #gluster
15:22 mattapp__ joined #gluster
15:26 mattap___ joined #gluster
15:27 plarsen joined #gluster
15:27 bala joined #gluster
15:33 bsaggy joined #gluster
15:33 wushudoin joined #gluster
15:36 mattapp__ joined #gluster
15:37 gmcwhistler joined #gluster
15:45 mattappe_ joined #gluster
15:46 _pol joined #gluster
16:00 sroy__ joined #gluster
16:04 vpshastry joined #gluster
16:04 vpshastry left #gluster
16:05 jag3773 joined #gluster
16:06 gdavis331 Greetings all. Does anyone know what the contributing factors are to slow metadata operations such as traversing, listing and searching through directories?
16:08 _pol joined #gluster
16:08 neofob joined #gluster
16:09 daMaestro joined #gluster
16:11 mattapp__ joined #gluster
16:11 jag3773 joined #gluster
16:14 zerick joined #gluster
16:15 shyam joined #gluster
16:16 spechal_ joined #gluster
16:16 spechal_ Where/Who should I tell that http://gluster.org/community/documentation/index.php/Gluster_3.2:_Triggering_Self-Heal_on_Replicate needs updated?
16:16 glusterbot <http://goo.gl/WuhYWX> (at gluster.org)
16:17 spechal_ left #gluster
16:19 ababu joined #gluster
16:21 LoudNoises joined #gluster
16:22 abyss^ I started replace-brick and virtual machine failed... Now I can't start or abort replace brick, any help?
16:24 social is glusterd operating?
16:24 kkeithley_ spechal_: what needs updated?  FWIW, you could create an account and fix it yourself.
16:25 spechal_ joined #gluster
16:25 social abyss^: what version, + check all logs, brick and daemon logs, everything should be OK unless you didn't force/commit the OP
16:25 social abyss^: and if you had replica setup it should be also fine
16:26 spechal_ I intentionally shut a node down and wrote a file to the volume.  When I brought the machine back up, there was no auto-healing.  When I tried to heal the volume from a client I got "Commit failed on XX.XX.XX.XX ..."  This is what was in the log: http://fpaste.org/59283/13862607/ ... anyone have any ideas on how to heal the volume?
16:26 glusterbot Title: #59283 Fedora Project Pastebin (at fpaste.org)
16:30 abyss^ social: I have gluster 3.3.1 and a lot of that things: 0-sa_bookshelf-replace-brick: transport.address-family not specified. Could not guess default value from (remote-host:(null) or transport.unix.connect-path:(null)) options
16:31 abyss^ social: I need do replace-brick again but I can't
16:32 dusmant joined #gluster
16:34 social abyss^: do you have testing instance somewhere?
16:34 abyss^ social: yes
16:35 social abyss^: can you reproduce the issue there?
16:35 abyss^ social: no, because it hard to do unexpected VM crash :)
16:36 social ^_^
16:41 social abyss^: well I don't have 3.3.1 anywhere around and it would take hours to spin some ;/
16:42 Eco_ joined #gluster
16:42 social abyss^: but I guess you have the replace brick operation in gluster volume status ? on test env I would just try restarting glusterd but dunno about production
16:44 shyam joined #gluster
16:46 mattapp__ joined #gluster
16:47 abyss^ social: I restarted gluster but it didn't help:(
16:48 nikkk is there a clean way to restart a server so the clients don't hang?
16:49 social abyss^: pls gluster volume status, gluster volume info if we can see them
16:49 anands joined #gluster
16:52 abyss^ social: http://pastebin.ca/2492919
16:52 glusterbot Title: pastebin - glust - post number 2492919 (at pastebin.ca)
16:52 abyss^ storage-1-saas is the server where the storage-gfs-3-prd have to be migrate
16:53 _polto_ joined #gluster
16:53 abyss^ if i do replace-brick abort then gluster behave strange and I can't do even command like heal info or so
16:54 social but clients are fine so far?
16:54 social if yes I would call the abort and wait, it would timeout anyway in 10min
16:54 abyss^ social: yes, clients are fine (I have Distributed-Replicate)
16:55 abyss^ social: As I said, if I do replace-brick abort then gluster behave strange and I can't do even command like heal info or so
16:55 social just wait while, don't kill it
16:56 spechal_ What is the problem, generally, when a volume heal fails with Commit failed to XX.XX.XX.XX?  http://fpaste.org/59283/13862607/ is what I have in the log
16:56 glusterbot Title: #59283 Fedora Project Pastebin (at fpaste.org)
16:56 abyss^ I don't kill it:) I wait till the end (it last about 5 min), then I get no result and gluster behave like above
16:56 social hmm that's timeout
16:57 social can you get logs from glusterd and brick? last 10min?
16:57 abyss^ Yes
16:58 _pol joined #gluster
16:58 abyss^ social: brick: http://pastebin.ca/2492920
16:58 glusterbot Title: pastebin - brick - post number 2492920 (at pastebin.ca)
16:59 abyss^ and gluster: http://pastebin.ca/2492921
16:59 glusterbot Title: pastebin - glu - post number 2492921 (at pastebin.ca)
16:59 abyss^ in bricks is still the same , repeat the same errors
17:00 social gluster peer status ?
17:02 abyss^ social: http://pastebin.ca/2492924
17:02 glusterbot Title: pastebin - Mine - post number 2492924 (at pastebin.ca)
17:02 ndk joined #gluster
17:03 abyss^ first is from storage-gfs3 and seconf from ydp-bsf-gfs-prd
17:03 abyss^ *second
17:06 davinder joined #gluster
17:07 social abyss^: I'd start poking someone more experienced than me :/
17:08 abyss^ social: OK. Thank you for your help. I hope somebody will answer...
17:09 social abyss^: last I would try cleanly restart glusterd on nodes one by one hoping that it'll forget about the op >.>
17:11 abyss^ Ok. I will check this too.
17:12 social abyss^: if I had something like this on production I did service glusterd stop; pkill gluster service glusterd start; gluster volume heal <vol>; find + stat on mount on client and after it passed and there was nothing in gluster volume heal <volume> info I moved to another node. But that's called shotgun debugging :)
17:18 rotbeard joined #gluster
17:19 dylan_ joined #gluster
17:20 zaitcev joined #gluster
17:22 hagarth joined #gluster
17:25 _polto_ joined #gluster
17:29 nikkk there's no professional services that the gluster foundation offers, right?  aka if implemented in an enterprise and shit goes wrong, there's nobody to call.
17:29 nikkk writing up something for managers here
17:29 spechal_ Is there commercial support for GlusterFS?  If a node dies and there isn't anything useful in the log and no one in IRC to help we are hosed, so we can't use it in our environment
17:30 nikkk right
17:30 nikkk that's pretty much it
17:30 spechal_ I guess that pretty much nixes us from using gluster due to lack of support
17:31 nikkk i think we can still use it however not for mission-critical purposes
17:31 Mo__ joined #gluster
17:31 Mo___ joined #gluster
17:31 spechal_ using it for something important/mission critical was the idea
17:31 spechal_ high availability and replicating NFS was something we really needed ... oh well
17:31 nikkk eg load balanced web servers - if one becomes unusable i can take it out of rotation while it's fixed
17:31 nikkk yeah, of course
17:31 nikkk that's what i want too :)
17:32 hagarth spechal_: Red Hat offers support for a product based on glusterfs - it is called Red Hat Storage
17:32 spechal_ :)
17:32 spechal_ Ok, I will check that out.  Thank you hagarth
17:32 nikkk as much as i dislike how it was done, i'm happy that there's mysql support, for example
17:32 spechal_ I think it just may be too soon for gluster, maybe not enough of a user base yet
17:33 mattappe_ joined #gluster
17:33 hagarth spechal_: You can probably check on the mailing lists if there are alternate vendors who provide support too.
17:34 nikkk honestly, i've been trying to find the right fit for us.. used redhat's gfs2 storage cluster, apache hadoop, nfs3/4, veritas cfs, and afs.. everything works in one way or another but gluster seems to be the best fit
17:34 spechal_ I really like it, I just have to break it a lot and in various ways to vet it for our production environment and fixing it when it breaks doesn't always go well ... i.e. I just turned off a node, wrote a file to the volume, turned back on the node and now I can't do anything with the node I restarted
17:34 nikkk is that a client or server?
17:34 spechal_ I will shoot an email out to the mailing list asking just that.  Thanks for the suggestion hagarth
17:35 spechal_ I wrote to the volume on via the client, the node I turned off was a replica brick
17:35 nikkk so far in my testing just restarting glusterd was enough to make everything happy
17:35 spechal_ now I get Commit failed on XX.XX.XX.XX (the node restarted) with nothing useful in the logs
17:35 nikkk but obviously my testing hasn't been comprehensive yet
17:36 spechal_ Were you able to heal after restarting a node?
17:36 spechal_ That's where I am at
17:36 nikkk it worked on it's own
17:36 nikkk no intervention
17:36 spechal_ *sigh* Glad it worked for you though
17:36 nikkk haha
17:37 nikkk yeah i'm thinking about this long and hard
17:39 spechal_ I think I may be making progress ... http://fpaste.org/59310/13862651/ ... when I get heal info it says my node (.230) is not connected, but when I do a peer status it reports as connected
17:39 glusterbot Title: #59310 Fedora Project Pastebin (at fpaste.org)
17:42 partner hmm while running for example fix-layout should i avoid doing any additional operations? i'd like to try replacing one brick as its not exactly sucking in files. then again the fix-layout will run for yet another 10+ days
17:43 partner 3.3.2 on debian wheezy.. i am still trying to figure out what to do with the previous issue where newly added brick on a new server is giving loads of MKNOD/UNLINK permission denied errors
17:46 _polto_ joined #gluster
17:48 spechal_ Does anyone know what this error means? I issued bluster volume start $VOL force ... volume start: gluster: failed: Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /gluster. Reason : No data available
17:49 semiosis selinux?
17:50 semiosis just a wild guess.  check the logs for more information.  the glusterd log & the brick log
18:02 spechal_ no selinux, http://fpaste.org/59316/66541138/ is what is in the log regarding this issue ... too new to bluster to really know what it means
18:02 glusterbot Title: #59316 Fedora Project Pastebin (at fpaste.org)
18:03 spechal_ that paste is from the brick log file
18:04 failshel_ joined #gluster
18:05 spechal_ Extended attribute trusted.glusterfs.volume-id is absent seems to be the crux of the issue ... going to try http://joejulian.name/blog/replacing-a-brick-on-glusterfs-340/
18:05 glusterbot <http://goo.gl/5zVKnT> (at joejulian.name)
18:05 jbd1 joined #gluster
18:07 social hagarth: http://build.gluster.org/job/regression/2813/consoleFull < I don't understand the failure
18:07 glusterbot <http://goo.gl/XvEB8o> (at build.gluster.org)
18:10 straylyon joined #gluster
18:11 hagarth social: looks like a spurious failure and the test runs fine on my laptop. The patch has been queued for regression again.
18:18 hagarth fileabug
18:18 japuzzo_ joined #gluster
18:18 hagarth @fileabug
18:18 glusterbot hagarth: Please file a bug at http://goo.gl/UUuCq
18:24 glusterbot New news from newglusterbugs: [Bug 854162] Feature - Support >64 node clusters in Hadoop plugin <http://goo.gl/wGBfzm>
18:25 _pol joined #gluster
18:27 _pol_ joined #gluster
18:32 TrDS how can the hostname of a peer (and its bricks) be renamed in gluster 3.3?
18:32 failshell joined #gluster
18:40 dbruhn nikkk, redhat has commercial support for gluster, it's packaged as Redhat Storage
18:43 nikkk gfs2 isn't gluster though, i though they were entirely different
18:43 nikkk right?
18:43 elyograg I have this testbed that I've been using to work on my gluster problems.  Is there any way to downgrade gluster from 3.4.1 to 3.3.1 without having to wipe the volume/bricks and re-copy 2.3TB of data to it?
18:43 nikkk i wasn't able to figure out how to get gfs2 running without the rest of their massive cluster software either
18:43 nikkk overly complex
18:43 dbruhn https://engage.redhat.com/content/gluster-whitepaper201310031039?sc_cid=70160000000c9ZlAAI&amp;s_kwcid=TC%7C19119%7Cgluster%20storage%7C%7CS%7Cb%7C34353266550&amp;gclid=CLW57sHembsCFQPNOgodjzoAqg
18:43 glusterbot <http://goo.gl/nKOAiH> (at engage.redhat.com)
18:44 elyograg it's a testbed, it doesn't have to stay online.
18:44 dbruhn if I remember right gfs2 was the previous project and gluster was intended to replace it in product placement
18:45 dbruhn the crap part about the redhat version of gluster and the nice part is it comes packaged as a whole OS installer, they have a limited set of supported hardware, that is growing.
18:47 _pol joined #gluster
18:53 spechal_ Ok, for anyone that cares or was following my conversation earlier, to recover a volume brick I had to create a new directory and use replace-brick ... stop gluster ... remove the .glusterfs directory from the old brick, remove the extended attributes on the old brick, start gluster, use replace-brick (changing the new brick to the old brick) and run a volume heal
18:56 elyograg is there any known bug with the rebalance status output?  I have six machines in my cluster (four with bricks) but the rebalance status is showing five of them as localhost and then one other.
18:57 elyograg http://fpaste.org/59339/38626985/
18:57 glusterbot Title: #59339 Fedora Project Pastebin (at fpaste.org)
18:57 elyograg this is 3.4.1
18:58 andreask joined #gluster
19:07 diegows joined #gluster
19:13 mkzero joined #gluster
19:16 h4idz joined #gluster
19:24 MacWinner joined #gluster
19:25 MacWinner hi guys.. I have 4 nodes and I want to do replica 2.. I can successfully gluster peer probe each of the hosts, however, the first host shows up as an IP instead of hostname when I do gluster peer status
19:25 MacWinner this causes an error when I try to do gluster volume create..
19:26 MacWinner the remote peers are complaining in the log that they do not have the hostname setup as a peer (even though they have the IP setup).. any ideas? I've tried to add the peers circularly
19:29 MacWinner seems like if I detach the IP from another node and reattach with the hostname i get closer
19:33 samppah MacWinner: can you try to do peer probe from second host to first one?
19:33 MacWinner samppah, i tried that.. didn't seem to update to hostname.. it just said the peer is already in the list
19:34 MacWinner samppah, i think this worked when I had a 2 server cluster before.. this time i have a 4 server cluster
19:34 samppah odd
19:34 MacWinner samppah, if I detach and then reattach, it seems to work…. however I have to do it a couple times
19:34 samppah MacWinner: what version you are using?
19:34 MacWinner because when I detach and reattach node1 from node3, then node3's IP appears in the list..
19:35 MacWinner latest.. 3.4.2 i believe
19:35 MacWinner lemme check
19:35 MacWinner 3.4.1
19:35 MacWinner from glusterfs-epel
19:39 samppah MacWinner: is it possible to detach every node and then probe first two nodes from each other before adding others?
19:39 rwheeler joined #gluster
19:39 MacWinner samppah, yeah.. I think I tried that, but will try again in just a little bit
19:41 samppah btw, does the hostnames used in gluster and in hosts match up?
19:41 MacWinner nope.. cause the host has 2 interfaces
19:42 MacWinner i'm using private IP which is GigE for all the gluster stuff.. however hte hostname of the box is based on public ip
19:42 MacWinner so there is node1 and node1-internal
19:42 MacWinner i'm doing all the command with the -internal hostname
19:43 samppah i'm not 100% sure but i think that it may cause problems
19:44 MacWinner samppah, even now that I have hte volume created and mounted?  can it cause problems going forward?
19:46 mattapp__ joined #gluster
19:49 samppah MacWinner: can't say for sure but i think i hit some issues with similar setup when doing rebalance
19:49 Comnenus left #gluster
19:52 MacWinner samppah, interesting.. i'll keep an eye on it
19:52 samppah yeah, probably other has something to say about this :)
19:55 rotbeard joined #gluster
19:57 plarsen joined #gluster
20:00 NigelSUK maybe i missed something, being new to glusterfs, but should the self heal daemon be running at all on a new setup or does that just kick in if it's actually performing a "heal" ?
20:01 JoeJulian NigelSUK: It's always running on any server as long as that server is part of an active volume.
20:01 MrNaviPa_ joined #gluster
20:02 NigelSUK hmm, seems to always says n/a on mine, and i do have a testvolume and 2 bricks running..
20:02 JoeJulian Under "Port"?
20:03 NigelSUK yup
20:04 JoeJulian That's because it doesn't use a port, it uses a named pipe.
20:04 JoeJulian Online, though, should be "Y".
20:04 NigelSUK ahh right, oki, panic over hehe, sorry im new to it and have much testing and reading left to do, it does have a Y in online yup.
20:05 JoeJulian No worries. :)
20:05 NigelSUK althought testing is somewhat being hampered at the moment due to amazons dodgy aws images causing some issues with sshd.
20:08 sarkis joined #gluster
20:09 sarkis hey guys trying to get glusterfs up on centos, i get this in the logs when i try to start glusterfsd
20:09 sarkis [glusterfsd.c:478:create_fuse_mount] 0-glusterfsd: Not a client process, not performing mount operation
20:12 JoeJulian sarkis: unless you REALLY know what you're doing, and have your own reasons for doing so, you won't be starting glusterfsd.
20:12 sarkis so its glusterd on centos?
20:13 sarkis sorry coming from debian it was just gluster-server
20:13 JoeJulian It's glusterd since 3.1.0
20:13 getup- joined #gluster
20:13 sarkis that was an older version too
20:13 sarkis so could be that..
20:13 sarkis cool ty
20:13 JoeJulian Ah, right... I blame semiosis.
20:13 semiosis huh what?
20:13 sarkis :P
20:13 samppah :D
20:13 sarkis might be glusterd on debian now too
20:13 sarkis i haven't looked
20:14 semiosis the process is called glusterd but the package is called glusterfs-server
20:14 semiosis on debian
20:14 sarkis oh right
20:14 semiosis the system service is also glusterfs-server, even though that spawns the glusterd process :/
20:15 kkeithley_ It's the same on RHEL and CentOS.
20:16 * JoeJulian sniffs kkeithley_'s coffee mug...
20:16 kkeithley_ er, the packaging, glusterfs-server has glusterd  The init script to start glusterd is <drumroll>glusterd</drumroll>
20:16 JoeJulian :)
20:17 NigelSUK JoeJulian, would that mean on rhel glusterfsd should NOT be started at boot time, and specifically just glusterd ?
20:17 kkeithley_ The only thing in my coffee is sweet-and-low, half-and-half, and hazelnut syrup.
20:17 JoeJulian NigelSUK: correct
20:18 NigelSUK i better check my service, i saw it in the list but i cant remember if it was marked to start, i hope not..
20:18 JoeJulian chkconfig glusterfsd off
20:18 JoeJulian Easy enough to make sure.
20:19 NigelSUK glusterfsd      0:off   1:off   2:on    3:on    4:on    5:on    6:off
20:19 NigelSUK thats not right
20:23 nikkk i was kinda wondering that too
20:24 nikkk glusterd starts glusterfsd though (i think) as well
20:26 NigelSUK okie, not sure how mine got turned on, i dont remember doing it specifically.
20:27 bgpepi joined #gluster
20:32 smellis anyone know if you're supposed to be able to create a glusterfs pool in libvirt with rhel 6.5? (centos)
20:32 smellis skipping the fuse mount of course
20:37 samppah smellis: http://libvirt.org/storage.html#StorageBackendGluster at least this gives me error "error: internal error unknown storage pool type gluster" :(
20:37 glusterbot <http://goo.gl/1IWsMZ> (at libvirt.org)
20:44 dneary joined #gluster
20:45 _pol joined #gluster
20:53 JonnyNomad joined #gluster
21:17 smellis yeah, that's where i'm at, was wondering if I was missing something
21:17 elyograg anyone know whether "localhost" showing up multiple times is a known problem with 3.4.1? http://fpaste.org/59339/38626985/ I have six hosts in my cluster.  This output shows six localhosts and then the sixth host.  The host that shows up last changes each time I do the status.
21:17 glusterbot Title: #59339 Fedora Project Pastebin (at fpaste.org)
21:18 smellis samppah: qemu tools seem to have support but not libvirt
21:18 sarkis which config does gluster set port 6996 as the listen port?
21:19 sarkis trying to telnet to ti and get connection refused on 6996
21:23 Eco_ joined #gluster
21:23 rotbeard joined #gluster
21:27 semiosis sarkis: what version of gluster do you have that is listening on 6996?
21:27 semiosis afaik no modern version (since 3.1.0) uses that port
21:27 semiosis see ,,(ports)
21:27 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
21:33 psyl0n joined #gluster
21:40 failshell joined #gluster
21:40 abyss^ I started replace-brick and virtual machine crashed... Now I can't start or abort replace brick. If I do abort then gluster behave very strange: can't do gluster heal info etc. When I do abort then after 5 min I get no results... Any help?
21:41 abyss^ s/then after 5 min I get no results/it finish about 5 min with no results/
21:41 glusterbot What abyss^ meant to say was: [histsearch it finish about 5 min with no results]
21:42 psyl0n_ joined #gluster
21:45 mattapp__ joined #gluster
21:48 samppah ®o
21:49 semiosis :O
21:49 matta____ joined #gluster
21:55 zerick joined #gluster
22:00 neofob left #gluster
22:04 gdavis331 Does anyone know what the contributing factors are to slow metadata operations such as traversing, listing and searching through directories?
22:12 sticky_afk joined #gluster
22:13 stickyboy joined #gluster
22:13 Excolo joined #gluster
22:13 mattapp__ joined #gluster
22:14 gdavis331 I understand that gluster doesn't really perform optimally in this area but what specifically are the things that will degrade or increase performance?
22:15 Excolo Ok, so Gluster started freaking out and causing problems on my production servers. One of the bricks, I killed gluster, and restarted glusterd, now I dont see glusterfsd on that server in the processes. The brick where gluster wasnt restarted shows its connected to the other, but gluster volume status shows the brick that was restarted as offline
22:15 Excolo any suggestions?
22:15 Excolo please? (of coarse this had to happen 15 minutes before I'm supposed to clock out, so now im stuck in the office fixing it)
22:17 dbruhn gdavis331, gluster does as well as the resources you put behind it. Things that you mentioned tend to be latency sensitive, so if you need those operations to speed up, you will want to decrease your latency, through disk, and network
22:18 sarkis is it required to start with 2 nodes, can i start with just 1?
22:18 sarkis i am trying to get 1 node up first, however, i run into I/O errors cannot access on the filesystem
22:18 sarkis is that because the 2nd nodes not reachable?
22:19 dbruhn Excolo, are you seeing anything in the logs indicating why the brick is not starting?
22:19 dbruhn something like device out of space, or something along those lines? maybe unable to read volfile
22:20 Excolo still looking, trying to grep through
22:20 dbruhn Excolo, look at your brick logs, they are probably more telling right now
22:21 dbruhn sarkis, it depends on what type of volume you want to use
22:22 Excolo [2013-12-05 21:56:05.403639] E [posix-handle.c:387:posix_handle_init] 0-myvolfs_v3-posix: Different dirs /export (128/65028) != /export/.glusterfs/00/00/00000000-0000-0000-0000-000000000001 (274455024/65028)
22:22 Excolo [2013-12-05 21:56:05.403672] E [posix.c:4348:init] 0-myvolfs_v3-posix: Posix handle setup failed
22:22 Excolo [2013-12-05 21:56:05.403690] E [xlator.c:385:xlator_init] 0-myvolfs_v3-posix: Initialization of volume 'myvolfs_v3-posix' failed, review your volfile again
22:23 psyl0n joined #gluster
22:24 gdavis331 @dbruhn i dont need them to be as fast as a flat traditional filesystem
22:24 mattappe_ joined #gluster
22:25 gdavis331 but these operations take quite a long time on my cluster
22:25 dbruhn Yeah it is always going to be slower.
22:26 dbruhn JoeJulian wrote a good article that kind of sums it up http://joejulian.name/blog/dont-get-stuck-micro-engineering-for-scale/
22:26 glusterbot <http://goo.gl/aebT1c> (at joejulian.name)
22:27 dbruhn Excolo, is the partition that /var/lib is in full on that server?
22:27 gdavis331 right and again I know that it's always going to be slower. I'm ok with trying to minimize this type of usage
22:27 Excolo no, var is part of the root partition which is 33% full
22:28 gdavis331 But is this expected?
22:28 gdavis331 time ls log | wc -l
22:28 gdavis331 671
22:28 gdavis331 real0m3.203s
22:28 dbruhn 1gb networking?
22:28 _amir joined #gluster
22:29 badone joined #gluster
22:29 Excolo dbruhn, not sure if that was directed towards me, but yes
22:29 gdavis331 2g  bonded
22:30 dbruhn sorry that was for gdavis331
22:33 failshel_ joined #gluster
22:33 dbruhn gdavis331, here is the output from some similar sized directories, the first is from a system with QDR infiniband and 7200 RPM SATA
22:33 dbruhn [root@ENTSNV03001EP 1]# time ls . | wc -l
22:33 dbruhn 1019
22:33 dbruhn real0m15.786s
22:33 dbruhn user0m0.003s
22:33 dbruhn sys0m0.007s
22:33 dbruhn [root@ENTSNV03001EP 1]#
22:33 dbruhn the second is QDR infiniband and 15k SAS
22:33 dbruhn [root@ENTSNV04001EP 1]# time ls . | wc -l
22:33 dbruhn 605
22:33 dbruhn real0m4.683s
22:33 dbruhn user0m0.004s
22:33 dbruhn sys0m0.003s
22:33 dbruhn [root@ENTSNV04001EP 1]#
22:34 dbruhn the 7200RPM SATA system is 6 servers, 12 bricks
22:34 dbruhn the SAS system is 10 servers, 20 bricks
22:35 gdavis331 my setup is currently on 2 servers with 12 1t bricks distributed mirror
22:35 gdavis331 and I want to expand that
22:36 dbruhn I run my system with 6 drive RAID5 configurations
22:36 gdavis331 what i'm wondering is more about design decisions
22:36 dbruhn this allows me to use the raid to improve the io of the subsystem
22:36 gdavis331 i currently use hw mirrors as the brick
22:36 Excolo dbruhn, because there seems to be something up with the volfile, would doing this: rsync -vvv -aHAX --whole-file --inplace root@server1:/export/myvol/ /export/ work?
22:37 gdavis331 so each has 12 hw mirrors, mirrored to another chassis
22:38 h4idz joined #gluster
22:38 dbruhn Excolo, it looks like you need to figure out which of your volume files is messed up on that brick
22:39 gdavis331 but does that design contribute to this issue? would using a raid 10 on the chassis and only having 2 bricks work better?
22:39 dbruhn Excolo, what kind of volume is this?
22:39 dbruhn gdavis331, are you in production with this system yet
22:40 mattappe_ joined #gluster
22:40 gdavis331 yes but it can be moved or modified
22:40 gdavis331 i have another 2 boxes to add to the cluster but kind of want to make these decisions first
22:40 Excolo ah, maybe this is it, .glusterfs/00/00/00000000-0000-0000-0000-000000000001 has nothing in it on the server with the problem, where as the other does
22:41 gdavis331 time ls archive | wc -l 43290
22:41 Excolo is it safe to just copy it over from one to the other?
22:41 gdavis331 real3m34.622s
22:41 dbruhn gdavis331, I can only suggest testing to be sure. There are some people who use single drives as bricks and a lot of them on here, and other guys who use RAID. I have never tested them side by side.
22:41 gdubreui joined #gluster
22:42 diegows joined #gluster
22:42 dbruhn Excolo, on the good server is .glusterfs/00/00/00000000-0000-0000-0000-000000000001 a link to like ../../..?
22:42 gdavis331 i figured that smaller bricks would give better concurrency and flexibility
22:43 Excolo no, thats odd, they're just all the files
22:44 Excolo oh wait, misunderstood
22:44 Excolo yes it is
22:44 dbruhn gdavis331, honestly if you have the extra servers there, I would test both, and maybe write up a blog entry on the subject, or provide feedback to the room. I am sure a lot of people in here would be interested in your findings
22:44 Excolo and its not on the problematic
22:44 dbruhn it should be a link matching what's going on, on the good one
22:45 gdavis331 but also even on you infiniband setup 4.5 secs to ls a dir with 600 files in it is very poor and I would imagine the issue would grow with larger file counts
22:46 gdavis331 I guess i'll just have to start running some side by sides
22:47 gdavis331 @dbruhn: thanks for you input. I'll let you know what i find
22:47 dbruhn gdavis331, thanks, I know the community loves feedback of any kind.
22:48 Excolo hell yes! brick is reporting back online, testing it out
22:49 dbruhn Excolo, congrats
22:49 dbruhn just in time to walk out the door for the night ;)
22:50 Excolo well not quite yet, still need to test everything, and wait at least a little bit to make sure it doesnt freak a bit
22:50 Excolo but at least I can watch dr who while I wait around for a bit to make sure it doesnt go out again ;)
22:50 Excolo (hey, im off the clock i can do what i like :-P )
22:50 eXeC64 left #gluster
22:50 Excolo thanks for the help!
23:02 B21956 joined #gluster
23:04 dbruhn joined #gluster
23:08 Excolo grrrr, dont get to go home yet
23:09 Excolo the static servers (the ones that deliver up the files) for some reason keep hanging trying to deliver any of the files, but I can ls them via command line
23:09 dbruhn what now?
23:09 dbruhn is the static server just a client?
23:09 Excolo just a client that runs nginx
23:10 Excolo *also runs
23:10 dbruhn can you unmount and remount it
23:10 Excolo yep, did several times
23:10 Excolo can ls images via command line and i dont get any errors, or timeouts, or anything
23:10 Excolo try and view it in browser, and i just get a spinning wheel
23:11 dbruhn that's weird
23:12 Excolo also, if I stop nginx it doesnt stop because it becomes a D process (waiting on disk), if I unmount gluster after that nginx stops
23:12 dbruhn can you reboot the server?
23:12 Excolo there are two of them, and I'd really rather not if I can avoid it
23:13 dbruhn if gluster is responding to the commands via the command line, it seems gluster is working, maybe something isn't happy in your app stack
23:14 Excolo oh, and another odd thing, every once and a while when I go through the entire process of restarting it, I get the image, and then reresh and same thing again
23:14 NigelSUK sounds like some kind of weird cache failure :|
23:15 zerick joined #gluster
23:17 Excolo GAH, thought I had it (killed the gluster process, stopped nginx, mounted gluster, cleanly unmounted, mounted again, restarted nginx) it worked about 10 times, then started hanging again
23:17 zerick joined #gluster
23:17 dbruhn is it a specific file ngix is hanging on?
23:17 NigelSUK Excolo, is this happening on "old" content, have you tried creating a few new files and seeing if the problem is affecting them ?
23:18 Excolo no I havent tried new content, i'll try
23:18 Excolo im testing with a specific image usually but ive tried a few
23:18 dbruhn Excolo what happens when you stat that specific file?
23:19 Excolo Size: 1268            Blocks: 3          IO Block: 131072 regular file
23:19 Excolo Device: 15h/21d Inode: 12557084297159652733  Links: 1
23:19 Excolo Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
23:19 Excolo Access: 2013-12-05 23:13:59.781220975 +0000
23:19 Excolo Modify: 2013-10-22 03:08:07.749659238 +0000
23:19 Excolo Change: 2013-12-05 23:08:21.924140156 +0000
23:19 dbruhn ok so you are not getting a split-brain error or anything like that from Gluster
23:21 Excolo nope, and it let me write a file to gluster without issue
23:21 Excolo going through the restart cycle to see if i can access it
23:22 Excolo got through a few more times with the new file, but still started hanging again
23:22 dbruhn Are your logs saying anything? from the gluster mnt log, or the ngix logs?
23:22 mattapp__ joined #gluster
23:23 Excolo ok, this is odd, another server that has the ability (but isnt a static server) but at the same datacenter is working fine
23:23 Excolo but two servers, that are at two different datacenters, are not
23:24 _BryanHm_ joined #gluster
23:27 Excolo not seeing anything in nginx logs (except for some file not founds, probably after I unmount gluster)
23:27 Excolo been looking through gluster, but nothing is jumping out at me
23:33 Excolo im at a loss, i have no fucking clue why this is happening
23:38 bala joined #gluster
23:40 Excolo may have figured it out, killed gluster, stopped nginx, mounted gluster, cleanly unmounted, mounted, started nginx all after taking the server out of production (I had left it in because none of the static servers were working, and I was hoping to get at least one working fast for less user impact)
23:45 NigelSUK time to head off here, hope you find a fix for your issues Excolo, nighty night :)
23:45 Excolo good night, i THINK *knock on wood* its getting better
23:46 Excolo my guess, the servers had an issue from inital gluster problem, our cdn kept trying to hit them up, and since they were still in production as soon as nginx came back online it was bombarding it with requests before it could get caught up (working theory)
23:56 bgpepi_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary