Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2014-02-02

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:00 purpleidea glusterbot: where are the semiosis ppas
00:00 purpleidea @ppa
00:00 glusterbot purpleidea: The official glusterfs packages for Ubuntu are available here: 3.3 stable: http://goo.gl/7ZTNY -- 3.4 stable: http://goo.gl/u33hy -- 3.5 QA: http://goo.gl/Odj95k
00:02 andrewklau joined #gluster
00:03 semiosis purpleidea: vagrant Q, if i run vagrant destroy, then next vagrant up will be totally clean right?
00:03 purpleidea semiosis: okay, i've got to head out shortly, but happy hacking, and expect some need to hack puppet-gluster for ubuntu. keep in mind the main git clone has the puppet code and the vagrant/gluster/puppet/modules/gluster/ dir has a clone of that
00:04 purpleidea semiosis: mostly, yeah
00:04 semiosis purpleidea: ok have a good one!  thx for all the help :)
00:05 purpleidea no worries. at some point i'll patch all the plumbing so that ubuntu is supported natively with puppet-gluster, but i'm lacking motivation at the moment.
00:05 purpleidea yw, later!
00:15 askb_ joined #gluster
00:20 askb_ joined #gluster
00:24 realdannys joined #gluster
00:31 semiosis @later tell purpleidea some notes re: vagrant... 1. if using latestest virtualbox, you *need* trusty, the saucy box has too old a virtualbox guest driver so shared folders dont work.
00:31 glusterbot semiosis: The operation succeeded.
00:32 semiosis @later tell purpleidea 2. set box hostname to vagrant-ubuntu-trusty-64.lan or puppet flips out. (these two are probably cause of my error earlier)
00:32 glusterbot semiosis: The operation succeeded.
00:33 jporterfield joined #gluster
00:47 RicardoSSP joined #gluster
00:47 RicardoSSP joined #gluster
00:52 jporterfield joined #gluster
01:14 xeed0 left #gluster
01:16 jag3773 joined #gluster
01:37 _zerick_ joined #gluster
01:47 mattapperson joined #gluster
01:49 vpshastry joined #gluster
01:52 mkzero joined #gluster
01:58 sarkis joined #gluster
02:05 semiosis purpleidea: not sure if i made any progress but i did clear a lot of puppet errors.  will pick this up another time, need a break
02:06 sarkis O_O
02:06 sarkis hi semiosis
02:06 semiosis two main themes seemed to be s/glusterd/glusterfs-server/ and cutting all the exported resources code (!)
02:06 semiosis sarkis: o/
02:06 * semiosis waves
02:10 semiosis sarkis: just saying hi?  or can I help you with something?  i'm about to split
02:12 sarkis oh ya sorry
02:12 sarkis just a gluster user thats a puppet user as well
02:13 semiosis do you use purpleidea's puppet-gluster module?
02:14 semiosis i was trying to use it to build a trivial volume in a vagrant vm to run integration tests against
02:16 sarkis yes
02:16 sarkis im about to use it for my day job stuffs
02:17 sarkis i haven't done that yet, but pretty proficient with vagrant, too.. stuck on something maybe i can help?
02:20 semiosis sarkis: the module depends on a puppetmaster with storeconfig/exported resources -- something i'm not going to have
02:20 sarkis oh crap
02:20 sarkis purpleideas does?
02:20 sarkis i didn't read that
02:21 sarkis is this a future release or?
02:22 semiosis so i removed all the exported resource code, but now it doesnt work... http://paste.ubuntu.com/6858821/
02:22 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
02:22 semiosis i'm using git master of purpleidea's module yeah
02:22 sarkis doh!
02:23 sarkis this is bad, that means i can't use it either :(
02:23 semiosis why?  any real production puppet steup should have a puppetmaster with storeconfigs... why doesnt yours?
02:24 semiosis my case is unusual, i'm building a trivial setup for software dev testing
02:24 sarkis hah, we are working on it... actually before i started working there we were on puppet 2.6
02:24 sarkis they adopted puppet 0.25.x and have a lot of mangled manifest
02:24 sarkis its a mess... trying to fix it all atm
02:25 semiosis hah yeah, my prod setup is still on puppet 2.7 & using my own gluster module that predates purpleidea's :D
02:25 semiosis and it's on gluster 3.1.7 lol
02:26 sarkis hah well i inherited gluster 3.0.x
02:26 sarkis and in the process of moving it to 3.4.1
02:26 sarkis or 3.4.2 now
02:26 sarkis :(
02:26 semiosis what distro?
02:26 sarkis so currently we were using .vol files..
02:26 sarkis moving to centos 6
02:27 sarkis but 3.0.x was on debian squeeze
02:27 semiosis thats a big jump... you might as well start fresh & copy in all your data... i cant imagine how to upgrade!
02:27 sarkis ya i did that
02:27 sarkis problem is i used the cli
02:27 semiosis thats good!
02:27 sarkis and was excited cause i can easily puppet it
02:27 sarkis but now i'm sad
02:27 semiosis ?
02:27 sarkis no storedconfigs for a bit here.. gotta figure out the interim
02:29 semiosis well my old unmaintained gluster puppet module is on github.  i dont recommend using it but it might give you some inspiration, until such time as you can use storeconfig... https://github.com/semiosis/puppet-gluster
02:29 glusterbot Title: semiosis/puppet-gluster · GitHub (at github.com)
02:29 sarkis ya
02:30 semiosis all it does is install & set up some monitoring
02:30 Jayunit100 joined #gluster
02:30 sarkis i don't see anywhere it says you need storeconfig on purpleideas
02:30 semiosis https://github.com/purpleidea/puppet-gluster/blob/master/manifests/host.pp#L131
02:30 glusterbot Title: puppet-gluster/manifests/host.pp at master · purpleidea/puppet-gluster · GitHub (at github.com)
02:30 semiosis for example
02:30 sarkis doh!
02:30 semiosis exported resources
02:38 sarkis well semiosis thank you !
02:39 sarkis i may end up taking a stab at un-coupling the exported resources but i need to figure out a better solution in the meanwhile so off i go ;(
02:40 semiosis later
02:41 robo joined #gluster
02:46 sarkis this one may work out
02:46 sarkis https://github.com/thias/puppet-glusterfs
02:46 glusterbot Title: thias/puppet-glusterfs · GitHub (at github.com)
02:46 * sarkis tries
02:52 mattappe_ joined #gluster
03:01 bala joined #gluster
03:31 jporterfield joined #gluster
03:39 bala joined #gluster
03:41 mattappe_ joined #gluster
04:03 bala joined #gluster
04:42 slappers joined #gluster
04:42 sulky_ joined #gluster
05:01 sarkis joined #gluster
05:11 ilbot3 joined #gluster
05:11 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
05:15 dblack joined #gluster
06:02 kshlm joined #gluster
06:19 bala joined #gluster
06:26 Jayunit100 joined #gluster
06:31 jporterfield joined #gluster
06:37 jporterfield joined #gluster
06:42 jporterfield joined #gluster
06:48 jporterfield joined #gluster
06:58 jporterfield joined #gluster
07:01 alias_willsmith joined #gluster
07:25 jporterfield joined #gluster
07:27 iksik joined #gluster
07:39 hagarth joined #gluster
07:42 jporterfield joined #gluster
08:12 purpleidea semiosis: some info: my puppet-gluster doesn't require exported resources at all, but it does add a certain level of automation that isn't possible without them. for a 1 host system, they aren't needed. also, for a 1 host system, gluster isn't very useful :P in any case, you can use my module to setup a full cluster without using exported resources.
08:13 purpleidea semiosis: iow, you shouldn't need to edit my module to do what you want. you might have to do something similar to: https://github.com/purpleidea/puppet-gluster/blob/master/examples/distributed-replicate-example.pp
08:13 glusterbot Title: puppet-gluster/examples/distributed-replicate-example.pp at master · purpleidea/puppet-gluster · GitHub (at github.com)
08:13 purpleidea @later tell sarkis have a look at 3:13am EST in the logs ^^^
08:13 glusterbot purpleidea: The operation succeeded.
08:14 purpleidea semiosis: also, while exported resources exist, you can still use this in a standalone, puppetmaster-less config. you'll see a warning when you run it saying that exported resources won't work, but puppet-gluster will still work.
08:24 hagarth joined #gluster
08:27 bala joined #gluster
08:49 jporterfield joined #gluster
08:59 ProT-0-TypE joined #gluster
09:10 ProT-O-TypE joined #gluster
09:59 bala joined #gluster
10:04 jporterfield joined #gluster
10:28 jporterfield joined #gluster
10:32 iksik_ joined #gluster
10:34 haomaiwang joined #gluster
10:35 ProT-0-TypE joined #gluster
10:36 solid_li1 joined #gluster
10:36 Nuxr0 joined #gluster
10:36 Peanut__ joined #gluster
10:38 swaT30_ joined #gluster
10:39 msvbhat_` joined #gluster
10:41 codex joined #gluster
10:41 dork joined #gluster
10:47 askb joined #gluster
10:47 harish joined #gluster
10:57 jporterfield joined #gluster
10:59 bala joined #gluster
11:11 jporterfield joined #gluster
11:19 mick271 joined #gluster
11:20 mick27 joined #gluster
11:23 mick271 joined #gluster
11:34 jporterfield joined #gluster
11:48 jporterfield joined #gluster
11:56 jporterfield joined #gluster
12:00 dneary joined #gluster
12:09 iksik joined #gluster
12:12 jporterfield joined #gluster
12:45 mattappe_ joined #gluster
13:07 harish joined #gluster
13:08 mattappe_ joined #gluster
13:10 baoboa joined #gluster
13:12 haomaiwa_ joined #gluster
13:18 haomai___ joined #gluster
13:21 realdannys joined #gluster
13:26 johnmilton joined #gluster
13:32 jporterfield joined #gluster
13:52 realdannys joined #gluster
13:59 jporterfield joined #gluster
14:05 jporterfield joined #gluster
14:08 qdk joined #gluster
14:23 DV joined #gluster
14:26 eastz0r joined #gluster
14:34 Eitia joined #gluster
14:34 Eitia Hello. Has anyone tested Gluster with Infiniband, and RDMA?
14:35 Eitia I'm getting better performance with NFS mount over IB than a gluster FUSE mount with rdma
14:35 Eitia like 3-4 better performance
14:36 Eitia dd if=/dev/zero of=foo bs=1024k count=1000 conv=fdatasync -> 171 MB/s over RDMA/FUSER and 600 MB/s over NFS/IB
14:37 Eitia version 3.5 qa beta2
14:42 leochill joined #gluster
14:50 jporterfield joined #gluster
15:04 jporterfield joined #gluster
15:12 jporterfield joined #gluster
15:32 mattappe_ joined #gluster
15:44 dneary joined #gluster
15:53 NuxRo joined #gluster
16:02 jporterfield joined #gluster
16:09 mattappe_ joined #gluster
16:11 mattap___ joined #gluster
16:17 mattappe_ joined #gluster
16:18 mattapperson joined #gluster
16:21 Ark_explorys joined #gluster
16:21 Ark_explorys Hello everyone.
16:22 Ark_explorys Anyone around that can assist with an interesting GlusterFS 3.3 volume issue?
16:22 dbruhn whats the issue?
16:25 Ark_explorys I had 2 servers, added 2 more and made replicated volumes into distributed replicated volumes. I rebuilt the first 2 servers and used the UUIDs of the old servers on the new. I did a rebalance and fix-layout on the gluster volumes after I got all 4 back up and running. The 2 old volumes that was in the original cluster will not gluster stop those volumes.
16:26 Ark_explorys 1 of the older volumes comes up as stopped, but when I go to delete it the volume says it is running and it will not delete.
16:27 dbruhn First question, no data on the system?
16:27 Ark_explorys I have backups I need
16:27 Ark_explorys i can copy all the information to HDFS and rebuild at worst case.
16:28 dbruhn You could kill all of the services and delete the volume files for the volumes that you're trying to get rid of potentially
16:28 Ark_explorys yes if I can manually remove the failed volumes that would be great
16:28 Ark_explorys could not find a lot of information on that
16:29 dbruhn joined #gluster
16:29 dbruhn sorry computer problems
16:30 Ark_explorys volume stop force does not work either because the volume is stopped, but then gluster says it cannot delete because it is still running.
16:30 Ark_explorys No problem thank you dbruhn
16:30 dbruhn /var/lob/glusterd
16:30 dbruhn is where the all of the volume files are
16:30 dbruhn before you start mucking around you'll want to back them all up
16:31 dbruhn also, one thing gluster does not like copying the UUID's between servers
16:31 Ark_explorys I was following gluster 3.2 documentation, might have been a mistake
16:32 dbruhn part of the reason it might be having an issue doing any volume manipulations is because the servers are non existent
16:32 dbruhn yeah, there were major changes from 3.2-3.3
16:32 dbruhn http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf
16:33 Ark_explorys i do a peer status and they all show up. If I have a brick online and the port does not show is that a worry as well?
16:33 dbruhn what does "gluster volume status" show
16:34 Ark_explorys Volume pdv-rdbms-backup is not started
16:34 dbruhn obviously trying to start it fails?
16:34 Ark_explorys sorry forget the port issue I brought up, all the bricks are showing port numbers for the volumes that are on at the moment.
16:35 Ark_explorys [root@gluster-backup-4 glusterd]# gluster volume start pdv-rdbms-backup
16:35 Ark_explorys Starting volume pdv-rdbms-backup has been unsuccessful
16:35 dbruhn agh ok, part of your issue might be a port conflict, gluster takes a new port for each brick it creates, if you have those overlapping in your config you will probably have issue s
16:35 dbruhn @ports
16:35 glusterbot dbruhn: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111 and 2049 since 3.4.
16:35 Ark_explorys so on the new servers  3/4 [2014-02-02 11:35:30.020333] I [client.c:2090:client_rpc_notify] 0-pdv-rdbms-backup-client-2: disconnected
16:36 dbruhn what is in this directory /var/lib/glusterd/vols
16:37 mattappe_ joined #gluster
16:37 dbruhn on all the servers
16:37 Ark_explorys drwxr-xr-x. 5 root root 4096 Jan  6 09:02 pdv-rdbms-backup
16:37 Ark_explorys drwxr-xr-x. 4 root root 4096 Jan 31 14:40 pdv-safe
16:37 Ark_explorys drwxr-xr-x. 5 root root 4096 Jan 24 20:54 prod-rdbms-backup
16:37 Ark_explorys drwxr-xr-x. 4 root root 4096 Jan 31 14:39 vertica-load
16:38 dbruhn ok, which volume is not working?
16:38 Ark_explorys i can tell you that the data inside  pdv-rdbms-backup the file timestamps are different on all the servers.
16:39 Ark_explorys this vol pdv-rdbms-backup does not have data I need. Just all the other volumes
16:40 Ark_explorys date stamps on the 2 servers I replaced are different
16:40 Ark_explorys if I stopped all services and deleted the vols info would gluster stop knowing about it?
16:40 dbruhn That's the idea I had, I can't promise anything
16:41 Ark_explorys Do you think I went down a bad road when I stole the old UUIDs and added them back to the servers I replaced?
16:41 dbruhn that's probably where it started
16:41 Ark_explorys I used the same hostnames if that makes and difference.
16:41 dbruhn I am not sure why you can't, I just know I have seen other say it's a bad idea to do it.
16:41 Ark_explorys Side note, when I add bricks should I just use IP addresses?
16:42 mattappe_ joined #gluster
16:42 dbruhn It's up to you, when a client connects gluster returns all of the servers to the client, so if DNS is working you can use hostnames, if not IP addresses are safer.
16:44 dbruhn The gluster fuse client actually connects to all of the servers in the cluster, thats how it is able to scale, and how it's a true distributed system.
16:47 mattapperson joined #gluster
16:47 Ark_explorys dbruhn:
16:48 Ark_explorys I am going to kick off a backup of all the volumes and then I will shut down and slay those files.
16:48 Ark_explorys Thank you for the information and I will report back here in several hours
16:48 dbruhn ok I might not be here, but someone might be if you need help.
16:49 Ark_explorys thanks for all the info as this has been a long and interesting ride
16:52 mick27 joined #gluster
16:52 mattappe_ joined #gluster
16:58 jporterfield joined #gluster
17:02 mattapperson joined #gluster
17:12 sarkis joined #gluster
17:15 mattappe_ joined #gluster
17:20 sarkis purpleidea: ah that makes sense, masterless will work, how about a puppetmaster with no puppetdb?
17:21 Napsty joined #gluster
17:23 Napsty Hi there. Doing some tests with Gluster and so far all tests successful. Hurray ^_^
17:24 Napsty I just have a question about creating a new gluster volume which was not answered through the official documentation (or I didnt find it). It seems to be necessary that the bricks must be mounted on the server, correct?
17:25 Napsty e.g. I tried to directly use a LV as brick paths (gluster volume create volxx replica 2 transport tcp server1:/dev/mapper/vg0-brick1 server2:/dev/mapper/vg0-brick1)
17:26 Napsty So just to make sure: Mount necessary?
17:28 dbruhn Napsty, they do need to be mounted directly
17:28 Napsty OK thanks a lot @dbruhn
17:28 dbruhn and they need to be a posix compliant file system
17:28 dbruhn xfs is suggested and the most widely tested
17:29 dbruhn ext4 is known to work, but not as widely used
17:29 Napsty yes tried to format the lv's as ext3 and then tried to create the gluster volumes. but "volume create: volxx: failed" occured.
17:29 dbruhn and there is a zfs writeup in the wiki
17:29 sarkis purpleidea: disregard last message, sorry, i am following along with : https://github.com/purpleidea/puppet-gluster/blob/master/examples/distributed-replicate-example.pp, looks like that won't be touching exported resources
17:31 Napsty That answers my question, thanks again
17:49 jikz joined #gluster
17:51 jporterfield joined #gluster
17:51 marvinc joined #gluster
18:04 jporterfield joined #gluster
18:23 TheDingy_ joined #gluster
18:41 social joined #gluster
18:42 social joined #gluster
18:46 ujjain joined #gluster
18:48 robo joined #gluster
19:02 rotbeard joined #gluster
19:12 social joined #gluster
19:13 jporterfield joined #gluster
19:31 andreask joined #gluster
19:50 edoceo joined #gluster
19:50 edoceo Is it possible to create a replica with only one brick?  I'll eventually have two, but for now I want to create one in a "degraded" mode or something
19:51 samppah edoceo: you can create volume with one brick and convert it to replica when you add next one :)
19:51 edoceo So, in that case I'd create a Distribute type, with only one right?
19:52 edoceo Then once my 2nd brick box arrives I can down the Distribute type and re-create as Replica?
19:52 samppah no need to tear it down or anything like that
19:52 samppah just gluster volume add-brick volName replica 2 server2:/newbrick
19:53 samppah and it will "convert" it to replicate
19:53 edoceo Really? I can just 'volumen add-brick?' and it will know
19:53 edoceo ahah, lag
19:56 edoceo Hm, I just deleted a volume and tried to re-create in the same directory and I'm told it's already part of a volume?
19:57 edoceo Can I just remove the '.glusterfs' directory ?
19:59 purpleidea sarkis: yeah, my puppet-gluster supports different methods for configuring your setup, depending on your needs... you can specify each component you want manually (hosts, bricks, volumes...) or you can use a combined type like gluster::simple to do this all for you.
19:59 purpleidea sarkis: the former doesn't need a puppetmaster or exported resources (although both are useful, and recommended) while the later needs both, but makes it very easy to configure.
20:00 purpleidea semiosis: ^^^ this might be information you're looking for from what i gather.
20:00 sarkis purpleidea: makes sense now, thank you very much
20:00 sarkis purpleidea: i am going to deploy a setup similar to the distributed reaplicated example you have
20:01 sarkis glad that i can use your module, was making plans to use something else until we get puppetdb up.. thanks for your response on this!
20:05 samppah edoceo: http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/ here is instructions what to do
20:05 glusterbot Title: GlusterFS: {path} or a prefix of it is already part of a volume (at joejulian.name)
20:05 edoceo thx
20:06 sarkis man i gotta say
20:06 sarkis gluster has come a long way since 3.0.x :)
20:06 sarkis so much better to work with, not sitting there wondering what the heck is really going on
20:10 spiekey joined #gluster
20:10 spiekey Hello!
20:10 glusterbot spiekey: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
20:11 spiekey i have 1 Gbit LInk between two nodes. MTU is 9000. iperf schows 1GBit. But with dd i only get an io speed of 300MBit?!
20:11 spiekey any idea where i could start digging?
20:17 NuxRo spiekey: replicated volume?
20:17 spiekey_ joined #gluster
20:18 NuxRo spiekey: replicated volume?
20:24 spiekey NuxRo: Type: Replicate
20:25 spiekey i just did some research and a lot of people say that glisters is known to have a poor write performance. Is that true or just a bad configuration?
20:28 spiekey sorry, laptop bat is empty
20:28 spiekey cu
20:29 NuxRo lol
20:29 sarkis haha
20:29 NuxRo cu
20:29 sarkis intereting guy
20:29 sarkis interesting*
20:29 NuxRo well, anyway, since he does replication, that means he's actually writing 2 x 300 Mbps
20:29 NuxRo add in some overhead, and voila, there's your 1 Gbps
20:30 Fresleven joined #gluster
20:38 purpleidea sarkis: np! good luck!
20:48 badone joined #gluster
20:55 mattappe_ joined #gluster
20:58 jporterfield joined #gluster
21:02 Eitia Hello. Has anyone tested Gluster with Infiniband, and RDMA?
21:03 flrichar joined #gluster
21:11 mattappe_ joined #gluster
21:16 mattappe_ joined #gluster
21:16 purpleidea Eitia: people have, yes.
21:26 jporterfield joined #gluster
21:30 mick272 joined #gluster
21:35 DV joined #gluster
21:42 MugginsM joined #gluster
21:51 andrewklau joined #gluster
21:53 mattappe_ joined #gluster
21:53 zmotok joined #gluster
21:55 zmotok hello everyone, I'm having a little problem here - after updating glusterfs from 3.4.1 to 3.4.2 (ScientificLinux 6.1 amd64), stopping glusterd glusterfsd and restarting them, all the bricks are offline; what could be the problem?
22:14 edoceo zmotok: do you just need to `gluster volume start $name` ?
22:14 zmotok edoceo: I did that, tells me:
22:14 zmotok gluster> volume start data-ext01
22:14 zmotok volume start: data-ext01: failed: Failed to get extended attribute trusted.glusterfs.volume-id for brick dir /external01. Reason : No data available
22:15 zmotok I'm checking and the glusterfsd isn't starting on any node..
22:15 zmotok all the bricks are offline
22:15 edoceo So, on each node are there errors when starting the gluster daemon?
22:16 badone gluster.org appears to be down
22:16 zmotok edoceo: I'm doing service glusterfsd start, then service glusterfsd status, and it's telling me it's stopped (and indeed there's no glusterfsd process alive)
22:17 edoceo do you have log files?
22:17 zmotok sure
22:17 zmotok which one should I upload (and where? paste.bin?)
22:24 mattappe_ joined #gluster
22:26 edoceo I think you should examine them, not me
22:26 edoceo I would start with something like `grep -ir gluster /var/log/`
22:26 edoceo Then, you might get lots of spew, and then you can examine the files that are shown
22:28 zmotok the logs show nothing as to why glusterfsd isn't starting
22:28 zmotok forcing it to use the .vol file I have been using since 2011 it just exits
22:28 zmotok ran it under strace, I see nothing weird, it's reading the file, spawning a child process then exits
22:31 badone zmotok: # ps auwwx|grep gluster
22:31 badone zmotok: anything?
22:32 zmotok badone: yes 1 glusterd, 2 glusterfs (--volfile-id gluster/nfs , --volfile-id gluster/glustershd)
22:33 badone grep " E " /var/log/glusterfs*
22:33 badone zmotok: ^ anything with today's date?
22:34 badone try /var/log/glusterfs/bricks as well
22:34 zmotok badone: nothing
22:34 zmotok not in bricks
22:35 badone zmotok: anything unusual in /var/log/messages or dnesg output?
22:35 badone *dmesg*
22:36 zmotok badone: some infiniband messages (ib0: Completion received during destruction of connection. Waiting)
22:36 zmotok nothing else of note
22:36 badone zmotok: try this...
22:37 badone zmotok: service glusterd stop
22:37 badone # pkill glusterfs
22:37 badone # ps auwwx|grep gluster
22:38 zmotok I did that multiple times already, I killed every gluster* process then restarted the services
22:38 zmotok didn't help
22:38 badone if there is nothing there then try "service glusterd start
22:39 zmotok all this results in the three processes I said are started
22:40 zmotok there's no glusterfsd process however, and the bricks are offline when looking with gluster volume status
22:41 badone zmotok: in /etc/init.d/glusterd change "GLUSTERD_OPTS="--pid-file=$PIDFILE ${GLUSTERD_OPTIONS}"" to "GLUSTERD_OPTS="--pid-file=$PIDFILE --log-level=DEBUG --log-file=/tmp/glusterd.log  ${GLUSTERD_OPTIONS}"
22:41 badone and try another restart and take a careful look at the logs
22:41 zmotok 1s
22:42 badone zmotok: sorry, take a careful look at /tmp/glusterd.log
22:45 zmotok badone: ok, I'm getting some errors there
22:45 zmotok 1s
22:45 badone zmotok: what sort of errors?
22:45 zmotok http://pastebin.com/aJ8NfaTG
22:45 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
22:46 badone zmotok: you are using infiniband?
22:47 badone zmotok: be back in 10
22:47 zmotok badone: yes
22:47 zmotok badone: ok, thank you
22:50 gdubreui joined #gluster
22:53 mattappe_ joined #gluster
22:53 zapotah joined #gluster
23:03 badone zmotok: looks like a problem with the RPC communications over IB
23:04 zmotok badone: and it wasn't there with gluster 3.4.1.. ?
23:04 mattapperson joined #gluster
23:05 jporterfield joined #gluster
23:05 badone zmotok: were the ib0 messages in dmesg there before you updgraded?
23:05 zmotok badone: yes
23:06 zmotok badone: it's due to the crappy qlogics
23:08 badone zmotok: afraid I don't know much about IB. Does everything else appear to be working okay in relation to the IB connections?
23:08 badone zmotok: could you temporarily test with normal IP?
23:08 zmotok badone: yes, I can spawn MPI processes running over IB only and they're fine
23:08 badone zmotok: understood
23:09 zmotok badone: I'll downgrade to gluster 3.4.1 and hope it fixes the problem
23:09 badone zmotok: sure, do you have any test systems?
23:10 zmotok badone: no, I did the big mistake of upgrading production (scientific work) clusters :-) in the hope to fix some problems we saw with 3.4.1 (input/output errors in replicated-distributed volumes)
23:10 zmotok and I need the systems back on tomorrow morning when new batches of jobs will start (got a break tonight as everything finished)
23:11 badone if 3.4.1 works then rdma.so may be borked, at least in your situation
23:11 badone zmotok: that is always the story :)
23:12 zmotok badone: unfortunately :(
23:12 badone zmotok: I'm here for at least another 6-7 hrs so let me know
23:13 zmotok badone: ok, thank you for your help!
23:13 badone zmotok: np
23:24 mattapperson joined #gluster
23:39 jporterfield joined #gluster
23:53 overclk joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary