Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-10-20

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:04 zhangjn joined #gluster
00:10 bennyturns joined #gluster
00:13 plarsen joined #gluster
00:15 dijuremo2 Guys, I tried the downgrade to 3.6.6 but I see: http://termbin.com/35j4
00:16 dijuremo2 Is my problem the wrong op-version since I downgraded from 3.7.3 yo 3.6.6 ?
00:20 cliluw joined #gluster
00:20 Dan39 joined #gluster
00:21 Dan39 hey bros
00:21 Dan39 i just had to downgrade 2 major releases, y aint it workin!?
00:22 Dan39 just kidding, hi :) just heard of gluster
00:23 Dan39 looking at site now, looks neat. i could probably use this :D
00:24 JoeJulian dijuremo2: edit /var/lib/glusterd/glusterd.info
00:25 JoeJulian Dan39: I like it. :D
00:25 dijuremo2 What should I set it to?
00:27 JoeJulian 30606
00:28 cliluw joined #gluster
00:28 Dan39 gluster can use most filesystems? i see the quickstart uses xfs...
00:28 dmnchild I have mine on ext4, just for learning purposes ;p
00:29 Dan39 i wonder if theres a reason for xfs
00:29 Dan39 if its in the quickstart i imagine there may be a reason to use it with gluster. unless the person who wrote it is just an xfs fanboy
00:30 Dan39 like even if i start using a different filesystem as my default, i would probably write tutorials using most common filesystem
00:30 dijuremo2 JoeJulian now the errors are:  0-management: wrong op-version (30606) retrieved
00:30 dijuremo2 and  0-management: Failed to restore op_version
00:32 JoeJulian 30603
00:33 JoeJulian That seems to be the latest op-version in 3.6.6 according to the source.
00:35 dmnchild Dan39: I read this, kinda says use whatever works for you but did have reasons: http://www.gluster.org/pipermail/glu​ster-users/2014-November/019436.html
00:35 glusterbot Title: [Gluster-users] Why is xfs recommended? (at www.gluster.org)
00:36 dijuremo2 glusterd starte, but now it thinks the bricks are offline
00:36 Dan39 ok good
00:36 Dan39 look dmnchild you can do more than lurk!
00:37 dmnchild lol. Still learning. but I do try and find answers before asking if I can ;p
00:37 Dan39 be the person that gets pissed when someone complains about problems when they try to downgrade a whole major release :P
00:37 dmnchild my next mission is geo replication which looks like it will be a pain
00:37 JoeJulian And xfs is the most common these days.
00:38 Dan39 most common *nix fs?
00:38 JoeJulian According to recent surveys.
00:38 Dan39 o wow, did ubuntu change to using xfs or something? :P
00:38 JoeJulian It has a cleaner code base, too, imho.
00:39 Dan39 i dont doubt it, i hear ext is a bit of a mess
00:39 dijuremo2 Locking failed on 10.0.1.6. Please check log file for details.
00:39 JoeJulian Don't tell Teddy I said that.
00:39 dijuremo2 The other node says the same about 10.0.1.7
00:39 Dan39 imo i think ext4 is the most common. surveys are easily tampered with ;)
00:40 JoeJulian No skin off my neck.
00:40 Dan39 but i have no idea tbh
00:40 Dan39 just from personal experience i see it used more
00:40 Dan39 i dont deal with enterprise stuff which is probably 90% of linux installations and they may all be using xfs and id never know
00:40 JoeJulian +1
00:41 Dan39 darn you logic
00:45 dijuremo2 JoeJulian, any ideas?
00:46 theron joined #gluster
00:46 dmnchild what did log show will be his first question?
00:47 klaxa joined #gluster
00:50 dijuremo2 http://termbin.com/tzur
00:52 vimal joined #gluster
00:53 JoeJulian Except that you keep starting and stopping the other servers, that log looks normal.
00:53 dijuremo2 The bricks show offline when I do a gluster v status
00:54 dijuremo2 Why would that be?
00:54 dijuremo2 I had never had that problem...
00:54 dijuremo2 I would just start glusterd and then gluster v status would show bricks online...
00:57 cliluw joined #gluster
01:02 zhangjn joined #gluster
01:11 EinstCrazy joined #gluster
01:15 dijuremo2 Which are the brick logs?
01:16 dijuremo2 Nevermind, found the folder..
01:17 dijuremo2 OK, seems like I am missing some files:   0-xlator: /usr/lib64/glusterfs/3.6.6​/xlator/features/trash.so: cannot open shared object file: No such file or directory
01:23 JoeJulian dijuremo2: find /var/lib/glusterd -name '*.vol' -exec mv {} {}.save
01:24 JoeJulian dijuremo2: glusterd --xlator-option *.upgrade=on -N
01:25 JoeJulian dijuremo2: That'll rebuild the vol files with features supported in 3.6.6
01:25 Lee1092 joined #gluster
01:28 rafi joined #gluster
01:30 haomaiwa_ joined #gluster
01:30 dijuremo2 I assume I have to stop glusterd, run the commands, then start, right?
01:30 JoeJulian right
01:37 dijuremo2 JoeJulian: that did it!!! you are the man!!!
01:40 harish joined #gluster
01:45 dijuremo2 More oddity, cannot mount the glusterfs volume
01:47 dijuremo2 So gluster volume status looks good now:
01:47 dijuremo2 http://termbin.com/tw6o
01:49 Humble joined #gluster
01:52 dijuremo2 If I do a heal info on export I get:     export: Not able to fetch volfile from glusterd
01:53 muneerse joined #gluster
02:03 dijuremo2 JoeJulian: seems like one more change is needed: /var/lib/glusterd/vols/export/info still shows:
02:03 dijuremo2 op-version=30702
02:03 dijuremo2 client-op-version=30702
02:07 jamesc joined #gluster
02:08 haomaiwang joined #gluster
02:26 kminooie left #gluster
02:30 dijuremo2 in gluster 3.7.x one can get all volume options with: gluster volume get VOL_NAME all
02:30 dijuremo2 How does one do that on 3.6.x ?
02:34 nangthang joined #gluster
02:45 halloo joined #gluster
02:49 haomaiwa_ joined #gluster
02:49 F2Knight joined #gluster
02:55 gem joined #gluster
03:01 JoeJulian dijuremo2: I assume you fixed the volume op-versions the same way we fixed glusterd. Your second question, "gluster volume info" shows settings that have changed from default. "gluster volume set help" has the defaults.
03:01 dijuremo2 I am all up and running now with 3.6.6
03:02 dijuremo2 Still need to see what happened to all the options I had in 3.7.x
03:02 dijuremo2 I did fix the op-version and that allowed me to mount that one volume: export
03:03 dijuremo2 JoeJulian, my options are still there... so I think time to head home... I could not have done it without your help. Thank you very much!!!
03:04 JoeJulian You're welcome.
03:04 dijuremo2 http://termbin.com/51mf
03:05 dijuremo2 Will have to play with more options to see if I can speed up my still present bad performance on slow files, but one battle at a time :)
03:07 haomaiwa_ joined #gluster
03:14 hagarth joined #gluster
03:17 julim joined #gluster
03:21 overclk joined #gluster
03:27 GB21 joined #gluster
03:29 TheSeven joined #gluster
03:32 sadbox joined #gluster
03:34 nbalacha joined #gluster
03:37 shubhendu joined #gluster
03:40 stickyboy joined #gluster
03:42 sakshi joined #gluster
03:44 rideh joined #gluster
03:45 GB21 joined #gluster
03:52 neha_ joined #gluster
03:52 [o__o] joined #gluster
03:58 maveric_amitc_ joined #gluster
04:02 itisravi joined #gluster
04:07 jatb joined #gluster
04:07 ramteid joined #gluster
04:20 ppai joined #gluster
04:21 deepakcs joined #gluster
04:27 jiffin joined #gluster
04:28 kshlm joined #gluster
04:42 rafi1 joined #gluster
04:48 RameshN joined #gluster
04:58 maveric_amitc_ joined #gluster
05:01 poornimag joined #gluster
05:09 aravindavk joined #gluster
05:10 ndarshan joined #gluster
05:10 hagarth joined #gluster
05:10 skoduri joined #gluster
05:12 Humble joined #gluster
05:17 poornimag joined #gluster
05:27 GB21 joined #gluster
05:28 Bhaskarakiran joined #gluster
05:29 vmallika joined #gluster
05:31 hgowtham joined #gluster
05:33 kanagaraj joined #gluster
05:34 haomaiwa_ joined #gluster
05:43 kotreshhr joined #gluster
05:45 ramky joined #gluster
05:47 itisravi joined #gluster
05:48 ashiq joined #gluster
05:49 atalur joined #gluster
05:49 raghu joined #gluster
05:50 Manikandan joined #gluster
05:58 Bhaskarakiran_ joined #gluster
06:05 dusmant joined #gluster
06:05 kdhananjay joined #gluster
06:06 gem joined #gluster
06:17 kanagaraj joined #gluster
06:18 jiffin1 joined #gluster
06:20 karnan joined #gluster
06:20 kanagaraj_ joined #gluster
06:25 jtux joined #gluster
06:28 kanagaraj joined #gluster
06:28 skoduri joined #gluster
06:33 poornimag joined #gluster
06:34 B21956 joined #gluster
06:37 kanagaraj joined #gluster
06:41 Philambdo joined #gluster
06:43 nangthang joined #gluster
06:53 kanagaraj joined #gluster
06:59 poornimag joined #gluster
07:00 raghu joined #gluster
07:01 sripathi joined #gluster
07:04 kanagaraj_ joined #gluster
07:05 lalatenduM joined #gluster
07:07 haomaiwa_ joined #gluster
07:07 mhulsman joined #gluster
07:07 jiffin1 joined #gluster
07:08 LebedevRI joined #gluster
07:09 [Enrico] joined #gluster
07:10 kanagaraj__ joined #gluster
07:23 poornimag joined #gluster
07:37 haomaiwa_ joined #gluster
07:38 fsimonce joined #gluster
07:44 maveric_amitc_ joined #gluster
07:45 kanagaraj_ joined #gluster
07:46 kdhananjay joined #gluster
07:48 kanagaraj joined #gluster
07:49 kshlm joined #gluster
07:51 Humble joined #gluster
07:52 kanagaraj joined #gluster
07:54 maveric_amitc_ joined #gluster
07:57 Sunghost joined #gluster
08:03 ctria joined #gluster
08:04 Ru57y joined #gluster
08:07 Trefex joined #gluster
08:11 RayTrace_ joined #gluster
08:11 jwd joined #gluster
08:18 Trefex joined #gluster
08:24 mlhamburg1 joined #gluster
08:36 arcolife joined #gluster
08:38 mhulsman1 joined #gluster
08:45 spalai joined #gluster
08:47 RayTrace_ joined #gluster
08:49 LebedevRI joined #gluster
09:07 KennethDejonghe joined #gluster
09:21 tdasilva joined #gluster
09:26 neha_ joined #gluster
09:28 vmallika joined #gluster
09:36 maveric_amitc_ joined #gluster
09:38 stickyboy joined #gluster
09:38 skoduri joined #gluster
09:40 auzty joined #gluster
09:40 al joined #gluster
09:54 rehunted joined #gluster
09:58 rjoseph joined #gluster
10:00 Manikandan joined #gluster
10:01 neha_ joined #gluster
10:02 yawkat joined #gluster
10:17 shyam joined #gluster
10:19 poornimag joined #gluster
10:19 raghu joined #gluster
10:27 overclk joined #gluster
10:31 kotreshhr1 joined #gluster
10:39 Slashman joined #gluster
10:40 deniszh joined #gluster
10:43 RayTrace_ joined #gluster
10:46 overclk joined #gluster
10:48 hagarth joined #gluster
10:49 dusmant joined #gluster
10:51 maveric_amitc_ joined #gluster
11:03 Manikandan joined #gluster
11:23 firemanxbr joined #gluster
11:25 overclk joined #gluster
11:26 kotreshhr joined #gluster
11:34 DV joined #gluster
11:47 DV joined #gluster
12:00 ekuric joined #gluster
12:00 kotreshhr joined #gluster
12:02 kxseven joined #gluster
12:11 drankis joined #gluster
12:13 Trefex joined #gluster
12:17 vmallika joined #gluster
12:21 spalai left #gluster
12:25 unclemarc joined #gluster
12:26 EinstCrazy joined #gluster
12:28 kkeithley joined #gluster
12:35 plarsen joined #gluster
12:36 kdhananjay joined #gluster
12:37 haomaiwang joined #gluster
12:47 overclk joined #gluster
12:48 aravindavk joined #gluster
12:52 kovshenin joined #gluster
12:56 monotek joined #gluster
13:00 Arrfab joined #gluster
13:01 Bhaskarakiran joined #gluster
13:09 maveric_amitc_ joined #gluster
13:09 GB21 joined #gluster
13:12 mpietersen joined #gluster
13:19 mjrosenb joined #gluster
13:20 zhangjn joined #gluster
13:21 zhangjn joined #gluster
13:22 zhangjn joined #gluster
13:22 kovshenin joined #gluster
13:23 zhangjn joined #gluster
13:28 hamiller joined #gluster
13:28 dgandhi joined #gluster
13:30 overclk joined #gluster
13:33 chirino joined #gluster
13:40 dlambrig left #gluster
13:42 sripathi joined #gluster
13:44 bennyturns joined #gluster
13:48 overclk joined #gluster
13:50 ivan_rossi joined #gluster
13:50 muneerse2 joined #gluster
13:52 cuqa joined #gluster
13:54 cuqa joined #gluster
13:58 neha_ joined #gluster
13:58 RameshN joined #gluster
13:58 zhangjn joined #gluster
13:59 zhangjn joined #gluster
13:59 ivan_rossi left #gluster
14:00 halloo joined #gluster
14:00 zhangjn joined #gluster
14:00 ivan_rossi joined #gluster
14:01 overclk_ joined #gluster
14:05 jamesc joined #gluster
14:08 harish joined #gluster
14:13 haomaiwang joined #gluster
14:13 7GHABGKAE joined #gluster
14:14 David_Varghese joined #gluster
14:22 rwheeler joined #gluster
14:25 Peppard joined #gluster
14:25 monotek1 joined #gluster
14:25 bennyturns ll
14:26 cabillman joined #gluster
14:30 RayTrace_ joined #gluster
14:31 al joined #gluster
14:38 rafi joined #gluster
14:39 Trefex joined #gluster
14:40 maserati joined #gluster
14:42 rafi1 joined #gluster
14:42 nbalacha joined #gluster
14:44 haomaiwang joined #gluster
14:44 bluenemo joined #gluster
14:45 rafi joined #gluster
14:48 arcolife joined #gluster
14:48 haomaiw__ joined #gluster
14:55 overclk joined #gluster
15:01 haomaiwang joined #gluster
15:03 ira joined #gluster
15:03 cholcombe joined #gluster
15:06 firemanxbr joined #gluster
15:06 haomaiwa_ joined #gluster
15:08 firemanxbr joined #gluster
15:09 firemanxbr joined #gluster
15:10 adamaN joined #gluster
15:17 kotreshhr left #gluster
15:17 jiffin joined #gluster
15:25 plarsen joined #gluster
15:29 rwheeler joined #gluster
15:38 stickyboy joined #gluster
15:40 doekia joined #gluster
15:58 wehde joined #gluster
15:59 wehde I need to move a windows fileserver over to gluster. Should I store the files in a flat file like vmdk or should i put all the files straight onto the gluster volume?
16:01 haomaiwa_ joined #gluster
16:03 overclk joined #gluster
16:06 halloo joined #gluster
16:11 a_ta joined #gluster
16:11 wehde what works better with gluster... a lot of small files or one giant file?
16:12 csim one giant
16:16 bennyturns joined #gluster
16:17 kshlm joined #gluster
16:19 nage joined #gluster
16:20 drankis joined #gluster
16:22 wehde i need an oppinion... we have 9TB of data stored in millions of little files (excel, word, etc)... should i run a windows VM on top of gluster and house the data there or should i run a samba vm on top of gluster that connects back to the underlying gluster volume and houses the data there?
16:25 wehde the downside that i see with the windows vm solution is that it's not very expandable for big data (you would end up creating a lot of data store vmdk's) and then attaching them to the vm. plus your vmdk file will never be able to be larger than the size of your smallest brick
16:26 wehde i've also noticed that vm's with large hard drisks tend to freeze up when gluster goes to replicate them...
16:28 JoeJulian It all depends on your use case and how you build your system.
16:28 JoeJulian @small files
16:28 glusterbot JoeJulian: See http://joejulian.name/blog/nfs-mount-for-glusterf​s-gives-better-read-performance-for-small-files/
16:29 JoeJulian I really should make a better factoid.
16:29 JoeJulian @forget small files
16:29 glusterbot JoeJulian: The operation succeeded.
16:32 wehde JoeJulian, love your blogs btw
16:33 haomai___ joined #gluster
16:33 JoeJulian @learn small files as When a replicated file is open, a check is made against each replica to see that the replica that's going to handle your fd isn't stale. This requires an extra RTT before the fd is open. On a small file, this RTT is a greater percentage the time needed to open, read, close the file that it is on a larger file. That's true for metadata operations too, which is why it can seem so expensive. Take this in to consideration when
16:33 JoeJulian designing your system.
16:33 glusterbot JoeJulian: The operation succeeded.
16:34 JoeJulian @small files
16:34 glusterbot JoeJulian: When a replicated file is open, a check is made against each replica to see that the replica that's going to handle your fd isn't stale. This requires an extra RTT before the fd is open. On a small file, this RTT is a greater percentage the time needed to open, read, close the file that it is on a larger file. That's true for metadata operations too, which is why it can seem so
16:34 glusterbot JoeJulian: expensive. Take this in to consideration when
16:34 JoeJulian @forget small files
16:34 glusterbot JoeJulian: The operation succeeded.
16:34 JoeJulian @learn small files as When a replicated file is open, a check is made against each replica to see that the replica that's going to handle your fd isn't stale. This requires an extra RTT before the fd is open. On a small file, this RTT is a greater percentage the time needed to open, read, close the file that it is on a larger file. That's true for metadata operations too, which is why it can seem so expensive.
16:34 glusterbot JoeJulian: The operation succeeded.
16:35 JoeJulian Thanks. I need to get back to that blog, I've just had no time.
16:36 hagarth joined #gluster
16:38 wehde That's the unfortunate downfall of opensource software... everyone has day jobs.
16:39 wehde JoeJulian, Do you see any issues running samba on a vm on top of a gluster volume and then having that vm connect back via nfs to store all the data?
16:40 unclemarc joined #gluster
16:41 JoeJulian If I were going to do that, I wouldn't bother with samba. Just mount nfs in windows. Use ganesha for the nfs head.
16:42 wehde I'm not as familiar with nfs in windows... would that allow me to still use active directory for permissions?
16:42 kshlm joined #gluster
16:47 JoeJulian No idea. I keep my distance from windows as much as possible.
16:49 wehde I wish i could... Unfortunately this company would die if they did have microsoft embedded everywhere...
16:50 wehde I dont thing nfs on windows would work
16:50 wehde I guess im stuck with samba4 and it's AD integration
16:52 wehde nfs-ganesha doesn't appear to be a stable release yet either
16:52 wehde https://access.redhat.com/documentation/​en-US/Red_Hat_Storage/3/html/Administrat​ion_Guide/sect-NFS.html#sect-NFS_Ganesha
16:52 glusterbot Title: 7.3. NFS (at access.redhat.com)
16:52 haomaiwa_ joined #gluster
16:53 JoeJulian Red Hat doesn't recommend a lot of things.
16:55 Trefex joined #gluster
16:55 overclk joined #gluster
16:57 wehde does gluster need anything special setup for nic bonding?
16:57 JoeJulian no
16:57 wehde i have 1Gb/s network with replica 3 but the max write i get is roughly 300Mbps
16:58 wehde each machine has two 1Gb/s nics
16:58 JoeJulian So roughly 1/3 of your total bandwidth... hmmm....
16:58 JoeJulian Like it's writing to all three replica.
16:58 JoeJulian Mostly because it is. :)
16:59 wehde but shouldn't it be utilizing essentially 2Gb/s
16:59 JoeJulian Which bonding method did you use?
17:00 wehde /etc/network/interfaces
17:00 wehde bridge_ports eth0 eth1
17:01 haomaiwa_ joined #gluster
17:02 Rapture joined #gluster
17:04 Rapture joined #gluster
17:05 JoeJulian I don't think that's bonding, though I'm no ubuntu expert. It seems to me that's just a bridge. I'm surprised it doesn't just cause a routing loop.
17:06 shubhendu joined #gluster
17:08 wehde i'll have to double check the configs after lunch
17:09 firemanxbr joined #gluster
17:10 deniszh joined #gluster
17:11 kshlm joined #gluster
17:17 jiffin JoeJulian: just out of curiosity, did u ever try pNFS over gluster volumes??
17:17 ivan_rossi left #gluster
17:18 JoeJulian I haven't. I only have 2 windows clients, and they're at home.
17:18 JoeJulian corosync/pacemaker just seemed like overkill.
17:19 JoeJulian (not that I don't do overkill at home)
17:20 jiffin JoeJulian: k
17:33 tomatto joined #gluster
17:35 rafi joined #gluster
17:38 rafi joined #gluster
17:42 calavera joined #gluster
17:45 rafi joined #gluster
17:48 rafi joined #gluster
17:52 rafi joined #gluster
17:54 rafi joined #gluster
18:01 haomaiwa_ joined #gluster
18:05 kovshenin joined #gluster
18:12 badone joined #gluster
18:51 theron joined #gluster
18:51 jbrooks joined #gluster
19:12 jobewan joined #gluster
19:16 unclemarc joined #gluster
19:18 halloo joined #gluster
19:21 thoht_ joined #gluster
19:21 thoht_ hi
19:21 glusterbot thoht_: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
19:22 thoht_ i just added a new brick to my volume (previously replica 1); so now it is a replica 2
19:22 thoht_ but the data on node2 are empty
19:22 thoht_ i proceeded like this: gluster volume add-brick  share_storage replica 2 node2:/shared/gluster/brick
19:22 thoht_ so gluster volume info shows me now that i m indeed in replica2 but the migration from node1 to node2 never happened
19:23 thoht_ i m trying to trigger it with : gluster volume replace-brick share_storage node1:/shared/gluster/brick node2:/shared/gluster/brick commit force
19:23 thoht_ but it returned volume replace-brick: failed: Pre Validation failed on devnix-virt-master02. Brick: node2:/shared/gluster/brick not available. Brick may be containing or be contained by an existing brick
19:23 JoeJulian what?!?! NO!
19:23 JoeJulian Oh good.
19:23 thoht_ why no ?
19:23 JoeJulian too many words.
19:23 JoeJulian first...
19:24 JoeJulian Check the self-heal logs in /var/log/glusterfs/glustershd.log
19:25 JoeJulian Once we figure out your replication issue, we'll come back to why you don't replace one brick with another brick in the same volume.
19:25 thoht_ JoeJulian: do you mean the option ?  option self-heal-daemon enable
19:25 thoht_ (i did a "grep heal /var/log/glusterfs/glustershd.log")
19:27 thoht_ JoeJulian: https://paste.ee/p/jnfIS
19:27 glusterbot Title: Paste.ee - View paste jnfIS (at paste.ee)
19:28 thoht_ JoeJulian: when i do a "gluster pool list"; the node2 has another name; is it bad ?
19:28 JoeJulian No
19:29 thoht_ what should i seek in the log exactly ? there is no reference to "heal"
19:29 Pupeno joined #gluster
19:29 Pupeno joined #gluster
19:29 JoeJulian So grepping for heal in the self-heal daemon log probably isn't that useful. You wanted to know why it wasn't healing. You should read through that log and look for problems.
19:29 thoht_ i can see this: 0-share_storage-client-1: disconnected from share_storage-client-1. Client process will keep trying to connect to glusterd until brick's port is available
19:30 thoht_ it looks scary
19:30 JoeJulian One hint, though, is that they're tagged with their error level, " E " would be an error.
19:30 thoht_ jermudgeon: oh thanks, i found this:  failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
19:31 thoht_ that s the only one E in node1
19:31 thoht_ same error on node2
19:31 mlhamburg1 Hi JoeJulian :) We talked on friday about a reconnection-problem. I filed a bug... https://bugzilla.redhat.co​m/show_bug.cgi?id=1272940
19:31 glusterbot Bug 1272940: high, unspecified, ---, bugs, NEW , Shd can't reconnect after ping-timeout (error in polling loop; invalid argument: this->private)
19:32 thoht_ jermudgeon: does it sounds bad ?
19:32 JoeJulian Thanks. I want to follow along with that, I'll add myself to the CC list.
19:33 JoeJulian I don't think jermudgeon pays much attention to this channel.
19:34 thoht_ sorry JoeJulian , it is a typo; it was for you :P
19:34 JoeJulian thoht_: So, did you follow the instructions it suggests?
19:34 thoht_ JoeJulian: yes captain; on both node
19:34 theron_ joined #gluster
19:35 thoht_ i can see the Brick node2 is N for column "onlin"
19:35 JoeJulian Does it show the port it's listening on?
19:35 thoht_ it is N/A
19:35 JoeJulian Are both servers on the same version?
19:35 JoeJulian Oh, N/A? Is the volume started?
19:36 thoht_ both 3.7.5
19:36 thoht_ JoeJulian: for brick1 it is started
19:36 thoht_ i mean i didn't do anything on node2 apart adding the brick with the command previously pasted
19:38 thoht_ JoeJulian: i did like that: https://paste.ee/p/IWQBw
19:38 glusterbot Title: Paste.ee - View paste IWQBw (at paste.ee)
19:38 JoeJulian Ok, yep. It's not started.
19:38 thoht_ JoeJulian: but i didn't see anything to start in doc or i miss it ?
19:39 thoht_ gluster volume start ? that's it ?
19:39 JoeJulian No, it should have started. Something unexpected must have gone wrong.
19:39 JoeJulian Any chance you rebooted node2 after you added the brick?
19:39 monotek1 joined #gluster
19:39 thoht_ it is a fresh install; i just installed the centos7
19:39 thoht_ no i didn't
19:40 JoeJulian Damn, my usual "guess right the first time" isn't working today.
19:41 JoeJulian Well, let's just try "gluster volume start share_storage force" on node2
19:42 thoht_ JoeJulian: succeeded; and now i ca nsee the TCP Port and Y for Online
19:42 JoeJulian excellent.
19:42 thoht_ but the folder is still empty :)
19:43 JoeJulian It'll get there.
19:44 thoht_ JoeJulian: https://paste.ee/p/g1Gh3
19:44 glusterbot Title: Paste.ee - View paste g1Gh3 (at paste.ee)
19:44 thoht_ this is the gstatus status
19:45 thoht_ anything to do to force the sync ?
19:46 thoht_ there is also this msg: Server and Client lk-version numbers are not same, reopening the fds
19:46 glusterbot thoht_: This is normal behavior and can safely be ignored.
19:47 thoht_ JoeJulian: ok it is good
19:47 thoht_ it is synching !!
19:47 thoht_ OMG you are a GOD JoeJulian !
19:47 thoht_ thanks :)
19:48 JoeJulian Hehe, you're welcome. :)
19:49 thoht_ JoeJulian: anyway to speedup the sync. ? any voodoo parameters ?
19:50 JoeJulian How much faster than your network do you want it to go?
19:51 haomaiwa_ joined #gluster
19:51 JoeJulian You could build a quantum teleportation network interface.
19:51 JoeJulian reduce your RTT to 0. That would be awesome!
19:51 thoht_ o_O
19:52 theron joined #gluster
19:52 thoht_ round trip time ?
19:53 thoht_ oh shit
19:53 JoeJulian right
19:53 thoht_ i do want the quantum teleporation nic
19:55 JoeJulian 3 continent replication with no latency. I can't wait.
19:55 thoht_ maybe we could have a DC on the moon for acting as a PRA
19:56 thoht_ you know; if the hearth is destroyed by a nuclear weappon; we need continuous service for facebook ...
19:56 JoeJulian hehe
19:57 thoht_ so now; i can add an arbitrer to my replica ?
19:57 thoht_ i got a third device but in another DC
19:58 calavera joined #gluster
19:59 plarsen joined #gluster
20:01 Gill joined #gluster
20:03 JoeJulian That would not be a good idea. If the link between DC's dies, you're one step closer to losing quorum, and you don't have an SLA for that link.
20:04 thoht_ JoeJulian: so replica2 is better than having an arbitrer ?
20:05 JoeJulian Your call. Depends on your requirements.
20:06 JoeJulian You could hang a Raspberry Pi in your DC as an arbitor. :D
20:06 Trefex joined #gluster
20:07 thoht_ JoeJulian: lol; in my case, the arbitrer would be a server 100mb but with no raid and weak cpu ability
20:09 JoeJulian The concern over the arbitor is the network connection. If it goes down or times out, you could potentially get a false positive.
20:09 jobewan joined #gluster
20:09 wehde joined #gluster
20:10 DV joined #gluster
20:23 a_ta_ joined #gluster
20:26 halloo joined #gluster
20:27 _maserati_ joined #gluster
20:40 Pupeno joined #gluster
20:40 Pupeno joined #gluster
20:52 Gill_ joined #gluster
20:54 wehde joined #gluster
20:58 JoeJulian Well... zfs as bricks was a terrible idea. Now I've got to juggle my available space at home so I can move everything back to xfs.
21:05 thoht_ JoeJulian: https://paste.ee/p/9LV2D
21:05 glusterbot Title: Paste.ee - View paste 9LV2D (at paste.ee)
21:05 thoht_ is it normal too see some undergoing heal on brick1 whereas the synchro didn't finish against node 2 ?
21:06 JoeJulian looking good
21:06 JoeJulian yep
21:06 arcolife joined #gluster
21:06 thoht_ it is so slow
21:06 thoht_ i had to interrupt the network for a couple of second between the 2 bricks and now it is not doing anything
21:07 thoht_ the file size are not growing on brick2 and only 2GB/250GB has been sync
21:07 thoht_ should i do something ?
21:07 a_ta_ left #gluster
21:08 JoeJulian Unless you changed ping-timeout, a couple of seconds shouldn't have had any effect.
21:08 thoht_ according to doc; it is 42 sec for ping-timeout
21:09 thoht_ JoeJulian: but why it is not doing anything anymore then ?
21:09 thoht_ logs show :  Subvolume 'share_storage-client-1' came back up; going online.
21:10 JoeJulian check "gluster volume heal share_storage statistics"
21:10 JoeJulian Interesting. So the client did see the brick go away for more than 42 seconds.
21:11 thoht_ https://paste.ee/p/Ivl0g
21:11 glusterbot Title: Paste.ee - View paste Ivl0g (at paste.ee)
21:11 thoht_ disconnected from share_storage-client-1. Client process will keep try
21:11 thoht_ ing to connect to glusterd until brick's port is available
21:11 thoht_ at 19h03
21:12 thoht_ and back online at  19:41:46
21:12 thoht_ strange
21:12 JoeJulian So according to that, it ended the crawl with nothing to heal successfully.
21:12 thoht_ what could i do to help him ?
21:12 JoeJulian Not sure how that's possible with stuff showing in heal info.
21:13 thoht_ the 2 files on heal info are 2 VM image not finished to sync
21:13 JoeJulian See if statistics shows the same thing on the other server.
21:14 thoht_ https://paste.ee/p/4Eews
21:14 glusterbot Title: Paste.ee - View paste 4Eews (at paste.ee)
21:14 thoht_ No. of heal failed entries: 9
21:14 JoeJulian Not most recently.
21:15 JoeJulian Oh, they're separated by brick
21:15 thoht_ separated ?
21:16 JoeJulian Or did you write that?
21:16 JoeJulian line 99 and 100
21:17 thoht_ Crawl statistics for brick no 1 ?
21:19 thoht_ i dont get you
21:19 thoht_ should i do : gluster volume replace-brick share_storage node1:/shared/gluster/brick node2:/shared/gluster/brick commit force ?
21:19 JoeJulian Did you type lines 99 and 100, or were they part of the output?
21:19 thoht_ there were part of the output
21:19 JoeJulian Are you just pulling my chain?
21:20 thoht_ i didn't type anything
21:20 thoht_ it is just a copy paste of the command you told me :)
21:20 JoeJulian Why would you want to do a replace-brick? Are you trying to retire node1:/shared/gluster/brick?
21:20 thoht_ nooo
21:20 JoeJulian But that's what that would do.
21:20 thoht_ i thought it will replace node2 with data of node1
21:21 JoeJulian It would replace node1:/shared/gluster/brick in your volume definition with node2:/shared/gluster/brick (which is already part of the volume which will luckily make that command fail).
21:22 thoht_ JoeJulian: i read on http://www.gluster.org/community/docu​mentation/index.php/Managing_Volumes
21:22 thoht_ For example, to migrate the data in server3:/exp3 to server5:/exp5 in test-volume:
21:22 thoht_ gluster volume replace-brick test-volume server3:/exp3  server5:exp5 start
21:22 JoeJulian *migrate* not *replicate*
21:22 thoht_ ohh
21:23 thoht_ i thought it was a new gluster terminology for replication :D
21:23 thoht_ hopefuly it failed
21:23 calavera joined #gluster
21:24 thoht_ so for the last 30mn, the sync is broken
21:24 thoht_ :(
21:24 JoeJulian Check the log
21:24 thoht_ should i restart glusterd on brick2 .? or any action i could do ?
21:24 thoht_ Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=share_storage --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
21:25 thoht_ samba ... sorry
21:25 JoeJulian Check glustershd.log on node1
21:26 thoht_ the log stop at  19:41:
21:26 thoht_ nothing more
21:26 thoht_ https://paste.ee/p/gBVlW
21:26 glusterbot Title: Paste.ee - View paste gBVlW (at paste.ee)
21:26 JoeJulian Is /var/log full? Does pgrep -f glustershd produce a result?
21:27 thoht_ yes i got the pid and var/log is not full
21:30 thoht_ readv on /var/run/gluster/2ae01a16e46​55d76acfde29cafe13876.socket failed (Invalid argument)
21:30 thoht_ i restarted gluster on node2 and now i got this log in loop
21:31 thoht_ Brick node2:/shared/gluster/brick           N/A       N/A        N       N/A
21:31 thoht_ brick offline
21:31 JoeJulian Check the brick log for that brick
21:32 thoht_ 0-glustershd: Ignore failed connection attempt on /var/run/gluster/5e289e228eae​8c72afd6ff5dd45baf95.socket, (No such file or directory)
21:32 thoht_ 0-share_storage-replicate-0: Subvolume 'share_storage-client-1' came back up; going online.
21:33 thoht_ i did gluster volume start share_storage force
21:33 thoht_ so it came back
21:35 thoht_ JoeJulian: can i stop gluster on node2; run a SCP of all the data; and start gluster back ?
21:37 thoht_ /dev/mapper/vg-share_gluster   300G    4,2G  296G   2% /shared/gluster
21:37 thoht_ localhost:/share_storage       300G    211G   90G  71% /share_storage_gluster
21:37 thoht_ data are crazy
21:38 thoht_ it is the same volume both; 1 local and secondly mounted through glusterfs type
21:39 stickyboy joined #gluster
21:47 thoht_ oh it is syncing now
21:48 JoeJulian thoht_: no, you won't have the correct metadata and you'll break your volume.
21:48 thoht_ JoeJulian: rsync can t keep the attributes ?
21:49 JoeJulian Sure it can. So then node2 has the attributes for node1.
21:49 thoht_ omg
21:49 JoeJulian instead of the attributes for node2.
21:49 thoht_ not good either
21:49 thoht_ totaly crappy
21:49 JoeJulian clusters are hard.
21:49 thoht_ but it is syncing now
21:50 thoht_ i run a scp of the vm image to node 2 but to a new folder (/backup°
21:50 thoht_ and in same moment; it began to gluster synch
21:50 thoht_ amazing
21:50 JoeJulian hehe
21:52 calavera joined #gluster
21:54 Pupeno joined #gluster
21:55 a_ta joined #gluster
21:55 Trefex evening, i have a problem with GlusterFS on oVirt. I am trying to mount a glusterFS volume and get an mount error
21:55 a_ta left #gluster
21:55 Trefex here you can see the error: http://paste.fedoraproject.org/281670/45378150
21:56 glusterbot Title: #281670 Fedora Project Pastebin (at paste.fedoraproject.org)
21:56 Trefex does anybody know where i can start debugging?
21:58 Trefex this glusterfs 3.7.5
21:59 JoeJulian Trefex: Is your volume started?
22:00 thoht_ Trefex: i just wiped ovirt on my disks
22:00 thoht_ and installed WebVirtMgr
22:00 Trefex JoeJulian: http://paste.fedoraproject.org/281671/53784161 this is on the host with the glusterFS
22:00 JoeJulian I run openstack at home.
22:00 glusterbot Title: #281671 Fedora Project Pastebin (at paste.fedoraproject.org)
22:01 Trefex thoht_: well...
22:01 thoht_ i was crazy with the self hosted engine
22:01 JoeJulian And they're all the same version?
22:01 Trefex JoeJulian: no, the server is 3.6 it seems
22:02 Trefex thoht_: yeah i have that too at the moment, self-hosted with about 50 VMs
22:02 Trefex client is 3.7 though
22:02 JoeJulian server is < 3.6.3
22:03 JoeJulian It's missing the correct form of op-version.
22:03 thoht_ Trefex: i guess you have at least 3 nodes
22:03 Trefex glusterfs 3.6.3 built on Apr 23 2015 16:11:43
22:03 JoeJulian hmm
22:03 Trefex thoht_: nope, unfortunately we're not there yet, right now only 1 node
22:03 Trefex thoht_: we want to go for 3 nodes in couple weeks, with cdtb and all that crap
22:03 thoht_ ohw
22:04 JoeJulian You guys keep shooting down my easy answers today.
22:04 Trefex JoeJulian: sorry :(
22:04 thoht_ oops
22:07 Trefex thoht_: will look at this Web thingy, does it work with Foreman ?
22:07 Trefex JoeJulian: anything else I could try? :S
22:08 DV_ joined #gluster
22:08 thoht_ Trefex: noooo it is a very modest project
22:09 Trefex thoht_: oh ok
22:09 thoht_ on hypervisor; you install only libvirtd
22:09 thoht_ nothing else
22:09 thoht_ the web gui autodiscovers everything
22:09 thoht_ but it is modest
22:09 thoht_ i was fedup with ovirt and vdsm
22:10 thoht_ vdsClient -s 0 list <== poor us
22:10 Trefex wahhhh that generated a big list :)
22:11 Trefex 56 entries
22:11 thoht_ your VMs i presume :P
22:11 Trefex ye
22:11 thoht_ but i prefer to do : virsh list
22:11 Trefex so what can I do about my problem ? :(
22:11 thoht_ no idea
22:12 thoht_ ask JoeJulian ; he is the guru
22:12 Trefex he fell off his chair
22:13 Trefex what happens if i update my ovirt host?
22:13 Trefex all hell will break loose?
22:14 theron joined #gluster
22:18 gildub joined #gluster
22:20 JoeJulian You /should/ already have allow-insecure set, so it should be easy.
22:21 JoeJulian I'm dealing with other odd problems...
22:21 JoeJulian tcp       11      0 0.0.0.0:49167           0.0.0.0:*               LISTEN      -
22:22 JoeJulian Note which process is listening on that port....
22:22 JoeJulian That's *supposed* to be one of my bricks, but it can't open the port because the kernel won't shut it.
22:24 thoht_ JoeJulian: each time i stop glusterfsd on my node2; then when i start it again; the brick stays offline and i ve to force start it :/
22:25 JoeJulian I'm going to repeat my mantra... check the logs.
22:26 thoht_ Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=share_storage --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
22:26 thoht_ it is because of samba?
22:27 thoht_ i don t use it
22:28 JoeJulian I don't know if it's because of that. Is that what's in the brick log?
22:28 JoeJulian or glusterd log?
22:29 JoeJulian I wish I'd never started with zfs. Looked good at first, now: load average: 134.86, 106.88, 59.36
22:30 JoeJulian It'll hit 300 before I give up and reboot.
22:31 m0zes_ joined #gluster
22:40 a_ta joined #gluster
22:46 brian_ joined #gluster
22:48 brian_ gluster is only letting me add one of my peers to the storage pool
22:48 brian_ I tried detatching one, then it lets me attach the other
22:48 brian_ and I can flip them back and forth - any ideas as to why i can't add them at the same time:?
22:53 bennyturns joined #gluster
22:53 brian_ I simply get the following error on which ever node is the second node I attempt to add: peer probe: failed: Probe returned with Transport endpoint is not connected
22:57 JoeJulian brian_: How many peers total?
22:57 brian_ I have one master and 2 'slaves'
22:57 brian_ I can utilize 2 / 3 of them
22:57 JoeJulian So 3 peers (/me grumbles about terminology)
22:58 brian_ yes
22:58 JoeJulian Transport endpoint is not connected is a TCP issue. glusterd isn't listening, or is firewalled.
22:58 JoeJulian Or the hostname isn't resolving correctly.
22:59 brian_ root@ip-10-0-10-35:/home/ubuntu# gluster --remote-host=10.0.10.59 peer status
22:59 brian_ Number of Peers: 1
22:59 brian_ i am able to contact it
22:59 brian_ via this command
22:59 JoeJulian And it has a peer.
22:59 brian_ correct
22:59 brian_ and if I dettach the peer that is connected
23:00 brian_ then I can attach the node that previously wouldn't attach
23:00 JoeJulian Are all your hostnames similarly named?
23:00 brian_ i'm using private IP addresses
23:00 brian_ in the 10.0.10.x rangne
23:01 JoeJulian Do all the hostnames resolve from each server?
23:01 JoeJulian (correctly)
23:01 JoeJulian Oh, you're saying you don't use hostnames.
23:02 halloo joined #gluster
23:02 JoeJulian Which two are currently peered?
23:03 JoeJulian Just identify them, you don't have to type long strings.
23:04 brian_ 10.0.10.59 (master), 10.0.10.35 (peer), 10.0.10.221 (error)
23:05 brian_ i am able to resolve IP's using the `host` command
23:06 JoeJulian Ok, so how are you trying to get "error" into the peer group?
23:06 brian_ yes
23:06 JoeJulian *how*
23:06 JoeJulian what command?
23:07 brian_ host <ipaddres>
23:07 brian_ root@ip-10-0-10-35:/home/ubuntu# host 10.0.10.59
23:07 brian_ 59.10.0.10.in-addr.arpa domain name pointer ip-10-0-10-59.ec2.internal.
23:08 JoeJulian You have a trusted pool consisting of 59 and 35. What command are you entering to attempt to add 221 to that pool?
23:09 brian_ root@ip-10-0-10-221:/home/ubuntu# gluster peer probe 10.0.10.59
23:09 brian_ peer probe: failed: Probe returned with Transport endpoint is not connected
23:09 JoeJulian Ah, that's why.
23:10 JoeJulian Note the word, "trusted" up there. When you've peered at least two computers, they form a trust. You cannot add yourself to that trust group, you have to be added from it.
23:10 JoeJulian probe 221 from either 59 or 35.
23:10 brian_ ah
23:11 JoeJulian I finally guessed one right today. :)
23:11 brian_ thanks!
23:28 haomaiwa_ joined #gluster
23:29 mlhamburg joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary