Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2017-08-05

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:27 rafi1 joined #gluster
00:56 vbellur joined #gluster
01:23 Lee1092 joined #gluster
01:50 kramdoss_ joined #gluster
01:52 ilbot3 joined #gluster
01:52 Topic for #gluster is now Gluster Community - https://www.gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - https://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
02:07 rdanter joined #gluster
02:15 shyam joined #gluster
02:43 msvbhat joined #gluster
02:54 baojg joined #gluster
03:26 kramdoss_ joined #gluster
03:44 msvbhat joined #gluster
04:19 atinm joined #gluster
04:46 kramdoss_ joined #gluster
04:52 h4rry joined #gluster
05:11 msvbhat joined #gluster
05:17 plarsen joined #gluster
05:26 msvbhat joined #gluster
05:33 baojg joined #gluster
05:41 msvbhat joined #gluster
05:42 rastar joined #gluster
05:44 AshishS joined #gluster
05:54 rastar joined #gluster
06:17 msvbhat joined #gluster
07:05 atinm joined #gluster
07:16 baojg joined #gluster
07:19 kramdoss_ joined #gluster
07:55 Lee1092 joined #gluster
08:02 kramdoss_ joined #gluster
08:09 nh2 joined #gluster
08:38 Wizek__ joined #gluster
10:20 rastar joined #gluster
10:34 kramdoss_ joined #gluster
10:54 h4rry joined #gluster
11:15 msvbhat joined #gluster
11:33 msvbhat_ joined #gluster
11:55 kramdoss_ joined #gluster
11:57 shyam joined #gluster
12:03 shaunm joined #gluster
12:22 msvbhat joined #gluster
12:56 jkroon joined #gluster
13:05 Acinonyx joined #gluster
13:15 atinm joined #gluster
13:38 msvbhat joined #gluster
13:39 Acinonyx joined #gluster
14:09 atinm joined #gluster
14:56 h4rry joined #gluster
15:03 pioto joined #gluster
15:31 iamtakingiteasy joined #gluster
15:32 iamtakingiteasy hi, i am planning to use glusterfs with kubernetes; but do i need heketi? If so, for what reasons? I've found lots of places saying heketi makes everthing convininet to kubernetes, but fail to grasp if it is needed for anything
15:55 http_GK1wmSU joined #gluster
15:55 http_GK1wmSU left #gluster
16:25 baojg joined #gluster
16:25 baojg joined #gluster
16:26 baojg joined #gluster
16:27 baojg joined #gluster
16:27 shyam joined #gluster
16:35 baber joined #gluster
16:47 susant joined #gluster
16:53 kusznir joined #gluster
16:53 kusznir Hi all: I had a cluster "crash" issue that I'm trying to recover from.  I've gotten most of it done, but there is some inconsistencies I still have that I'd like some help with.
16:54 kusznir First, I have a 3 node gluster (+ovirt) cluster, two replicas plus arbitrar.  They are ovirt1, ovirt2, and ovirt3, with 3 being the arbitrrar.  I took ovirt2 down for maintence, but at some point during that time, I did not realize ovirt3 also failed/went down.
16:55 kusznir All my ovirt nodes default to ovirt1 for their storage, so VMs were still running initially.  I ended up loosing all the gluster storage space on ovirt2 (hard drive repartition/replacement; was not able to get a full backup of gluster, but did get the OS fully backed up and restored).
16:57 kusznir I figured this wouldn't be a problem; the data should be restored from other nodes in the cluster.  I got all my partitions in order, lvm all set, etc, etc., then booted up the node.  I still was unaware of the fact that ovirt3 gluster was down at this time, and once ovirt2 booted, it took down the entire gluster storage cluster, and caused all VMs to be paused.
16:58 kusznir That's when I ran gluster volume status and saw only ovirt1 and ovirt2 listed, and for ports on ovirt2, it showed NA.  I started up ovirt3, and eventually got ovirt to recover and resume operations, but ovirt2 didn't respond.
16:58 kusznir I eventually found a document on rackspace that talked about xattr setting uuids copied from other servers, so I did that, and then ovirt2's gluster bricks came online (ports assigned), but stil were empty, and it didn't appear the self-heal started.
17:00 kusznir So I ran gluster volume heal <brickname> full, and that started the healing.  All was loooking good, except when it finished, two of the bricks storage utilzation didn't match ovirt1 (and ovirt1 and ovirt2 did match before all of this).
17:01 kusznir I can confirm through du -s on the actual gluster mounted filesystem that the larger number was correct, and ovirt2 was reporting the smaller number.  In one brick, the entire cluster reports space utilization (df -h) of the smaller, ovirt2 number, while another brick reports the (correct) larger number.
17:01 kusznir So at this point, I'm looking to figure out how to get the cluster truly consistent.
17:02 kusznir I'm also having problems getting ovirt to bring the storage domain online, it complains it cannot find the master domain, and lists some UUIDs.  Storage is mounted, and VMs are running.
17:06 sona joined #gluster
17:10 Acinonyx joined #gluster
17:25 msvbhat joined #gluster
17:28 baojg joined #gluster
17:29 shyam joined #gluster
17:32 kusznir So, if I stop gluster on ovirt2, then run heal info on the two non-fully-synced bricks, it shows a number of entries for ovirt1 and ovirt3 (and the number of entries are not the same).  But it doesn't appear to actually be doing anything.
17:33 kusznir As in, the number of entries just slowly goes UP, not down....
17:34 kusznir I launched a self heal full, but the number of entries keeps going up.....
17:34 kusznir I believe that ovirt1 has an "authoritative" set of data on it....
17:35 kusznir Oh, and split-brain info shows no entries.
17:45 plarsen joined #gluster
17:47 JoeJulian kusznir: wrt "master domain" That's an ovirt question. Gluster has no "master".
17:48 JoeJulian kusznir: Number of entries going up is normal. Let it finish and stop killing servers or it has to start the crawl all over again.
17:54 rafi joined #gluster
17:57 shyam joined #gluster
18:14 vladn joined #gluster
18:15 vladn i have an error when mounting use NFS ( portmap query failed: RPC: Program not registered , requested NFS version or transport protocol is not supported )
18:15 glusterbot vladn: make sure your volume is started. If you changed nfs.disable, restarting your volume is known to work.
18:15 vladn on a centos7
18:20 JoeJulian vladn: also, gluster's internal nfs is disabled by default now. ganesha-nfs is preferred.
18:29 baojg joined #gluster
19:09 kusznir JoeJulian: After well over an hour, the number of entries went from 66 to 99, and matches on both servers functioning.  I don't see much network traffic (or even high cpu load).  In fact, gluster is barely doing anything at all.  How can I tell if its actually working toward fixing the problem vs just watching dependencies grow as the filesystem is used?
19:09 kusznir In the past, whenever a heal is needed, it moves a lot of data and the numbers go down....
19:15 JoeJulian gluster volume heal $vol statistics
19:15 JoeJulian Or get a state dump and look for self-heal locks.
19:17 kusznir gluster volume heal .... shows all 0's except for the first "INDEX shows "no of entries healed: 2, and no of heal failed entries: 9.  That was from 10:24 (pdt).
19:17 kusznir how do I get a state dump and look for self-heal locks?
19:17 kraynor5b1 joined #gluster
19:18 kusznir btw: ovirt shows "20 min" for self-heal info, but has shown that for the past few hours.
19:30 baojg joined #gluster
19:42 sona joined #gluster
20:00 siel joined #gluster
20:05 vladn i have an error when mounting use NFS ( portmap query failed: RPC: Program not registered , requested NFS version or transport protocol is not supported )
20:05 glusterbot vladn: make sure your volume is started. If you changed nfs.disable, restarting your volume is known to work.
20:05 vladn on a centos7
20:23 jkroon joined #gluster
20:23 kusznir JoeJulian: Ok, so since the last conversation, on server 2, while gluster was completely stopped, I blew away all data (including .gluster and .shards) in the problem bricks, then restarted gluster and ran a heal full.
20:24 kusznir It moved a lot of data, but now seems to be stuck with only one entry different on each.
20:24 kusznir It also has far less space in use on those two than the other one.  The larger space also matches du -sh commands on the mounted gluster filesystem.
20:24 kusznir So it still seems like the filesystem is inconsistent.
20:25 kusznir Unless there's something else going on...(deduplication or something?)
20:33 jiffin joined #gluster
20:42 bwerthmann joined #gluster
20:46 jiffin joined #gluster
20:55 jiffin joined #gluster
20:58 h4rry joined #gluster
20:58 shyam joined #gluster
21:00 ^andrea^ joined #gluster
21:09 atrius joined #gluster
21:19 kusznir Ok, things went worse, unfortunately.  My VM infrastructure went down hard, and I'm trying to bring it back.  I'm finding that when I try and mount my gluster export in ovirt, I get: permissions settings on the specified storage do not allow access to the storage to vdsm user and kvm group.
21:19 kusznir I've checked the keys for the export, and the user and group IDs are set appropriately.
21:32 baojg joined #gluster
21:46 rdanter joined #gluster
21:50 d33pb00k-GK1wmSU joined #gluster
21:51 d33pb00k-GK1wmSU left #gluster
22:11 riyas joined #gluster
22:14 kusznir wow..finally think I have things laregely back to normal.  The one remaining issue is how my rebuilt host has smaller gluster storege utilization than the other full replica...I'd really feel a lot better if someone could at least answer why that is...
22:15 Acinonyx joined #gluster
22:41 shyam joined #gluster
23:33 baojg joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary