Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-08-10

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:08 masuberu hi
00:08 glusterbot masuberu: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
00:09 masuberu I have 10 Windows machines that writes 10 files per second to a NAS and I would like if gluster would fit in this scenario
00:09 masuberu 70% of those files are of size between 20 and 30Kbits
00:12 om joined #gluster
00:20 jdossey joined #gluster
00:22 Gambit15 masuberu, it would help if you stated what your objective was...
00:24 masuberu I work for medical research company and we have 10x Illumina HiSeqX machines that sequence human DNA and I need a storage for that. Now I want to reduce costs so I would like to setup something on commodity hardware and a replicated filesystem sounds like a good resiliancy solution for me
00:24 masuberu each of those sequencing machines runs jobs that takes 3 days
00:24 masuberu each job generates over 2.4 millions files which 70% of them are between 20 and 30Kb
00:26 masuberu on the other hand I have a HPC cluster which runs a pipeline that analyze the files created by the sequencing machines, this software is constantly reading and writes from the storage
00:26 masuberu at the moment I am using a Panasas
00:27 masuberu and I am having latency issues which means that the sequencing machines takes longer to write the files remotely hence our productivity drops dramatically for a few hours
00:35 Gambit15 masuberu, if you need fast I/O for small files, you'll need a bunch of nodes & a very good network, especially with replication
00:35 Gambit15 https://s3.amazonaws.com/aws001/guided_trek/Performance_in_a_Gluster_Systemv6F.pdf
00:36 masuberu each of my compute nodes have x2 26GB mellanox NICs and x10 2TB nvme drives, 512 GB RAM and x28 cores cpu
00:37 Gambit15 How many compute nodes?
00:37 masuberu I have 13 but I was thinking to use 5 for this, would that be enough?
00:38 masuberu I am planning to buy more nodes in this year
00:40 Gambit15 Unless you're using SSDs, I think you'll need more nodes. You haven't said what the I/O, although if it's data analysis stuff, that usually uses a good chunk
00:40 Gambit15 Check out the PDF I linked
00:44 Gambit15 For specialised use, you could also look into tuning your filesystem for small files. Larger stripe units & tuned inode sizes
00:45 Gambit15 Right, I'm out.
00:45 * Gambit15 tips hat
00:56 shdeng joined #gluster
01:06 muneerse joined #gluster
01:11 Alghost joined #gluster
01:17 B21956 joined #gluster
01:22 Lee1092 joined #gluster
01:47 ilbot3 joined #gluster
01:47 Topic for #gluster is now Gluster Community - http://gluster.org | Documentation - https://gluster.readthedocs.io/en/latest/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:47 Alghost_ joined #gluster
01:49 wadeholler joined #gluster
02:01 nishanth joined #gluster
02:04 om joined #gluster
02:11 Javezim joined #gluster
02:12 Javezim Hey all, has anyone seen an issue with Gluster where overrighting a file actually creates 2 files on the volume both with the same time/date and size?
02:12 Javezim If you then modify it and save it overwriting again it would create a 3rd file
02:19 masuberu hi
02:19 glusterbot masuberu: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
02:27 harish joined #gluster
03:03 Gambit15 joined #gluster
03:12 bkolden joined #gluster
03:14 JoeJulian Javezim: yes, but not since around 3.4.2 or so.
03:18 luis_silva joined #gluster
03:19 Javezim @JoeJulian We're on 3.7.11
03:19 magrawal joined #gluster
03:19 Javezim @JoeJulian Everything an overwrite is done it creates a new file, but then the old files all change their properties to match the new file
03:20 JoeJulian You're writing to a mounted volume, not a brick, right?
03:20 om joined #gluster
03:23 hagarth joined #gluster
03:31 nbalacha joined #gluster
04:01 itisravi joined #gluster
04:06 rastar joined #gluster
04:08 Javezim @JoeJulian Yeah
04:11 masuberu joined #gluster
04:15 rafi joined #gluster
04:17 karnan joined #gluster
04:19 shubhendu joined #gluster
04:29 aspandey joined #gluster
04:33 om joined #gluster
04:33 Javezim We've also updated gluster and samba vfs to 3.8.1 but have noticed decreases in performance, especially when accessing samba share. Anyone else noticed this?
04:35 poornimag joined #gluster
04:35 shubhendu joined #gluster
04:36 RameshN joined #gluster
04:45 karthik_ joined #gluster
04:48 Javezim Does anyone know what this is linking too?
04:48 Javezim http://paste.ubuntu.com/22883435/
04:48 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
04:48 Javezim Our directory is called /vault01mel/
04:49 Javezim why does it say /vault01me
04:51 sanoj joined #gluster
05:07 shubhendu joined #gluster
05:11 shubhendu joined #gluster
05:11 shubhendu joined #gluster
05:12 shubhendu joined #gluster
05:15 rastar_ joined #gluster
05:19 jiffin joined #gluster
05:26 Manikandan joined #gluster
05:30 Siavash_ joined #gluster
05:31 aravindavk joined #gluster
05:32 nehar joined #gluster
05:32 skoduri joined #gluster
05:36 mchangir joined #gluster
05:41 ashiq joined #gluster
05:44 rafi joined #gluster
05:46 poornimag joined #gluster
05:49 kotreshhr joined #gluster
05:51 Bhaskarakiran joined #gluster
05:51 atinm joined #gluster
05:53 Pupeno_ joined #gluster
05:54 jesk joined #gluster
05:55 _ndevos joined #gluster
05:55 _ndevos joined #gluster
05:56 prasanth joined #gluster
05:57 pdrakeweb joined #gluster
06:02 nishanth joined #gluster
06:03 aspandey joined #gluster
06:03 ramky joined #gluster
06:04 satya4ever joined #gluster
06:07 hgowtham joined #gluster
06:07 ankitraj joined #gluster
06:13 kdhananjay joined #gluster
06:13 msvbhat joined #gluster
06:14 karthik_ joined #gluster
06:16 prasanth joined #gluster
06:18 ashiq joined #gluster
06:19 jtux joined #gluster
06:25 atalur joined #gluster
06:29 ashiq joined #gluster
06:34 ppai joined #gluster
06:36 shubhendu_ joined #gluster
06:36 derjohn_mob joined #gluster
06:36 Muthu_ joined #gluster
06:42 Philambdo joined #gluster
06:48 Pupeno joined #gluster
06:49 devyani7_ joined #gluster
06:49 devyani7_ joined #gluster
06:50 gem joined #gluster
06:57 karthik__ joined #gluster
07:01 Javezim Okay have found that to be a bug with Samba logs - https://bugzilla.samba.org/show_bug.cgi?id=10987
07:01 glusterbot Bug 10987: medium, medium, ---, nalin, CLOSED RAWHIDE, MySQL Apache module
07:01 Javezim However we are still noticing a large drop of performance on 3.8.1 Samba VFS
07:01 Javezim In Comparison to 3.7.11
07:03 anoopcs Javezim, What's the Samba version?
07:04 Javezim @anoopcs
07:04 Javezim Version 4.3.9-Ubuntu
07:05 Javezim glusterfs 3.8.1 built on Jul  8 2016 18:14:17
07:05 anoopcs Javezim, I see that the upstream bug that you have mentioned has been reported against version 3.6
07:05 anoopcs Javezim, Do you still those log entries with 4.3?
07:05 anoopcs s/still those/still see those
07:08 anoopcs Javezim, Under which workload are you experiencing performance drop?
07:09 [diablo] joined #gluster
07:11 anil joined #gluster
07:12 Javezim @anoopcs Yeah its been spamming the logs all day
07:13 Javezim @anoopcs Basically just accessing a folder/file takes forever. We also run some scripts that check the files data (Date, Size, Name) and its processing just slow overall
07:14 Siavash_ joined #gluster
07:16 ppai joined #gluster
07:18 fsimonce joined #gluster
07:21 anoopcs poornimag, Are you aware of any specific changes(especially in libgfapi) between 3.7.11 and 3.8.1 that could have decreased the performance for the above use case specified above?
07:21 rastar joined #gluster
07:25 om joined #gluster
07:37 hackman joined #gluster
07:37 derjohn_mob joined #gluster
07:38 poornimag joined #gluster
07:44 Javezim We tried turning Mandatory Locking to Optimal
07:45 Siavash_ joined #gluster
07:47 jri joined #gluster
07:49 msvbhat joined #gluster
07:50 sanoj joined #gluster
07:51 Siavash_ joined #gluster
07:51 bkolden joined #gluster
07:52 hackman joined #gluster
08:00 arcolife joined #gluster
08:04 hackman joined #gluster
08:05 gem joined #gluster
08:06 shdeng joined #gluster
08:06 karthik_ joined #gluster
08:06 Siavash_ joined #gluster
08:14 harish_ joined #gluster
08:16 Javezim joined #gluster
08:17 deniszh joined #gluster
08:18 Siavash_ joined #gluster
08:19 hackman joined #gluster
08:21 anoopcs Javezim, Tuning mandatory-locking into different modes would be useful only when changes are made in vfs module for glusterfs in Samba based on support for the same in libgfapi for which a patch(https://review.gluster.org/#/c/11177/) is still under review.
08:21 glusterbot Title: Gerrit Code Review (at review.gluster.org)
08:23 pur joined #gluster
08:35 hackman joined #gluster
08:37 Javezim @anoopcs Then I have no idea why the VFS seems to be going to slow, prior to the weekend on 3.7.11 it was running great
08:40 atalur joined #gluster
08:41 Philambdo joined #gluster
08:41 itisravi joined #gluster
08:44 Philambdo joined #gluster
08:48 itisravi joined #gluster
08:51 itisravi joined #gluster
08:51 mchangir joined #gluster
08:53 ira joined #gluster
08:53 Slashman joined #gluster
08:57 Muthu_ joined #gluster
09:00 poornimag joined #gluster
09:01 social joined #gluster
09:19 rafi joined #gluster
09:19 jiffin1 joined #gluster
09:21 anoopcs Javezim, Me neither. I will inform if I can find something related.
09:22 rafi1 joined #gluster
09:32 rafi1 joined #gluster
09:34 Philambdo joined #gluster
09:36 arcolife joined #gluster
09:44 msvbhat joined #gluster
09:58 satya4ever joined #gluster
10:02 jri_ joined #gluster
10:03 msvbhat joined #gluster
10:05 kovshenin joined #gluster
10:14 jiffin1 joined #gluster
10:16 jri joined #gluster
10:23 rossdm joined #gluster
10:23 rossdm joined #gluster
10:26 hackman joined #gluster
10:28 B21956 joined #gluster
10:31 prasanth joined #gluster
10:37 Gnomethrower joined #gluster
10:48 mchangir joined #gluster
10:53 msvbhat joined #gluster
10:53 cvstealth joined #gluster
10:55 hackman joined #gluster
10:56 wadeholler joined #gluster
11:00 atrius joined #gluster
11:03 pur joined #gluster
11:08 ju5t joined #gluster
11:08 shdeng joined #gluster
11:09 satya4ever joined #gluster
11:13 prasanth joined #gluster
11:17 kshlm joined #gluster
11:22 poornimag joined #gluster
11:26 arcolife joined #gluster
11:26 Gnomethrower joined #gluster
11:28 social joined #gluster
11:28 jtux joined #gluster
11:30 mchangir joined #gluster
11:30 kshlm Weekly community meeting starts in 30 minutes in #gluster-meeting
11:44 Javezim #/join gluster-meeting
11:49 Gnomethrower joined #gluster
11:50 jtux joined #gluster
11:51 jiffin joined #gluster
11:56 johnmilton joined #gluster
11:57 social joined #gluster
12:00 karthik_ joined #gluster
12:02 Philambdo joined #gluster
12:05 social joined #gluster
12:13 shyam joined #gluster
12:21 B21956 left #gluster
12:31 nehar joined #gluster
12:34 harish_ joined #gluster
12:37 ju5t joined #gluster
12:46 Manikandan joined #gluster
12:50 Manikandan joined #gluster
12:51 karthik_ joined #gluster
12:51 aravindavk joined #gluster
12:54 nehar joined #gluster
12:55 luis_silva joined #gluster
12:57 unclemarc joined #gluster
13:06 social joined #gluster
13:07 chirino joined #gluster
13:09 shaunm joined #gluster
13:11 theron_ joined #gluster
13:14 derjohn_mob joined #gluster
13:19 ju5t joined #gluster
13:23 scuttle|afk joined #gluster
13:24 julim joined #gluster
13:28 mchangir joined #gluster
13:32 Javezim @anoopcs Am seeing this from one of te windows machines accessing the Samba Share
13:32 Javezim http://paste.ubuntu.com/22910816/
13:32 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:35 kovshenin joined #gluster
13:36 jiffin1 joined #gluster
13:42 Javezim Woa, just checked the glustefs-<share>.log
13:42 Javezim http://paste.ubuntu.com/22911531/
13:42 glusterbot Title: Ubuntu Pastebin (at paste.ubuntu.com)
13:42 Javezim Spammed with All subvolumes are down. Going offline until atleast one of them comes back up.
13:43 Javezim Gluster volume staus
13:43 Javezim Shows all bricks online
13:44 anoopcs Javezim, Check brick logs for any errors.
13:44 prasanth joined #gluster
13:46 anoopcs Javezim, Assuming you have set wide links = yes for share, it will be disabled when unix extensions = yes (which is the default)
13:47 anoopcs (unless you have set the overriding parameter allow insecure wide links = yes which is not recommended)
13:48 Javezim @anoopcs No errors in Bricks logs, Taking a look at smb.conf We have wide links = yes global but wide links = no for the folder. we also don't have unix extensions = yes specified
13:49 anoopcs Javezim, unix extensions is enabled by default.
13:50 anoopcs Javezim, What do you mean by "but wide links = no for the folder"?
13:51 aravindavk joined #gluster
13:52 rwheeler joined #gluster
13:53 muneerse joined #gluster
13:54 theron_ joined #gluster
13:54 Javezim like [global] is set wide links = yes, but [vault01mel] (With a path = /vault01mel) is set wide links = no
13:55 skylar joined #gluster
13:56 Javezim We hadn't changed any of this upgrading from 3.7.11 -> 3.8.1 either
13:57 muneerse2 joined #gluster
13:57 anoopcs Javezim, I was about to tell you the same
13:57 anoopcs Unless you have changed anything in smb.conf while upgrading it doesn't matter
14:00 Javezim @anoopcs this worries me though - http://paste.ubuntu.com/22911531/
14:01 anoopcs Javezim, Are you able to access the share now?
14:01 Javezim Yeah I can access it (slowly though)
14:02 anoopcs WHAT?
14:03 anoopcs Are you looking at the exact logs? That's weird..
14:03 anoopcs What do you have after those error entries?
14:03 shyam joined #gluster
14:04 hagarth joined #gluster
14:05 Javezim honestly its just spam of them
14:05 Javezim every second/few secons
14:10 bowhunter joined #gluster
14:10 muneerse joined #gluster
14:12 mchangir joined #gluster
14:14 muneerse2 joined #gluster
14:14 anoopcs Javezim, Those error entries doesn't seem right..
14:15 anoopcs Can you check whether you are loosing the connection between Samba server and bricks?
14:17 muneerse joined #gluster
14:20 harish_ joined #gluster
14:20 Philambdo1 joined #gluster
14:21 muneerse2 joined #gluster
14:37 nbalacha joined #gluster
14:39 eclay joined #gluster
14:43 eclay I originally had a 3 replica volume that I had some difficulties removing one brick from yesterday.  I was able to eventually get the brick removed but before I add a new one back I need some understanding of what the following could mean.  http://pastebin.ubuntu.com/22916240/
14:43 glusterbot Title: Ubuntu Pastebin (at pastebin.ubuntu.com)
14:43 skoduri joined #gluster
14:44 eclay gluster volume heal volume info shows 1 entry on on gluster server and 0 on the other.  It's been this way for about 20 hr.
14:45 eclay I would think that this means there is a file on one brick that isn't on the other or something hasn't replicated properly but need some help.
14:56 atinm joined #gluster
15:07 archit_ joined #gluster
15:11 wushudoin joined #gluster
15:28 jdossey joined #gluster
15:35 sankarshan joined #gluster
15:36 hagarth joined #gluster
15:48 Lee1092 joined #gluster
15:50 jdossey joined #gluster
15:51 nbalacha joined #gluster
15:54 kpease joined #gluster
15:58 theron joined #gluster
15:59 arcolife joined #gluster
16:07 jiffin joined #gluster
16:10 baojg joined #gluster
16:25 rastar joined #gluster
16:27 Gambit15 joined #gluster
16:27 kovshenin joined #gluster
16:32 gem joined #gluster
16:33 jiffin joined #gluster
16:37 msvbhat joined #gluster
16:41 shyam joined #gluster
16:42 theron joined #gluster
17:00 jbrooks_ joined #gluster
17:04 jdossey joined #gluster
17:08 theron joined #gluster
17:09 dnunez joined #gluster
17:17 jiffin joined #gluster
17:20 rafi joined #gluster
17:20 derjohn_mob joined #gluster
17:24 sputnik13 joined #gluster
17:26 Siavash_ joined #gluster
17:33 jdossey joined #gluster
17:34 msvbhat joined #gluster
17:38 Wizek joined #gluster
17:39 ashiq joined #gluster
17:43 shyam joined #gluster
17:46 dnunez joined #gluster
17:50 theron joined #gluster
17:50 theron joined #gluster
17:56 Siavash_ joined #gluster
18:01 jbrooks_ joined #gluster
18:09 mchangir joined #gluster
18:17 jbrooks joined #gluster
18:21 jiffin joined #gluster
18:27 hackman joined #gluster
18:27 jdossey joined #gluster
18:32 DV__ joined #gluster
18:42 Wizek_ joined #gluster
19:02 rastar joined #gluster
19:06 deniszh joined #gluster
19:09 Guest_84847 joined #gluster
19:09 Guest_84847 Allah is doing
19:09 Guest_84847 sun is not doing Allah is doing
19:09 Guest_84847 moon is not doing Allah is doing
19:09 Guest_84847 stars are not doing Allah is doing
19:10 Guest_84847 planets are not doing Allah is doing
19:10 Guest_84847 galaxies are not doing Allah is doing
19:10 Guest_84847 oceans are not doing Allah is doing
19:10 Guest_84847 mountains are not doing Allah is doing
19:10 Guest_84847 trees are not doing Allah is doing
19:10 Guest_84847 mom is not doing Allah is doing
19:10 Guest_84847 dad is not doing Allah is doing
19:11 Guest_84847 boss is not doing Allah is doing
19:11 Guest_84847 job is not doing Allah is doing
19:11 Guest_84847 dollar is not doing Allah is doing
19:11 deniszh left #gluster
19:11 Guest_84847 degree is not doing Allah is doing
19:11 Guest_84847 medicine is not doing Allah is doing
19:11 Guest_84847 customers are not doing Allah is doing
19:11 Guest_84847 you can not get a job without the permission of allah
19:12 Guest_84847 you can not get married without the permission of allah
19:12 Guest_84847 nobody can get angry at you without the permission of allah
19:12 Guest_84847 light is not doing Allah is doing
19:12 Guest_84847 fan is not doing Allah is doing
19:13 Guest_84847 businessess are not doing Allah is doing
19:13 Guest_84847 america is not doing Allah is doing
19:13 * ttkg waits for moderator to doing
19:13 derjohn_mob joined #gluster
19:14 post-factum JoeJulian: ^^
19:14 Guest_84847 fire can not burn without the permission of allah
19:14 post-factum JoeJulian: haaaaalllp!!1
19:15 Guest_84847 knife can not cut without the permission of allah
19:15 jesk my dick can not jerk of without allah doing
19:15 post-factum crap
19:16 Guest_84847 rulers are not doing Allah is doing
19:16 post-factum Guest_84847: tell me more
19:17 Guest_84847 governments are not doing Allah is doing
19:17 jesk kids arent raped, Allah is doing
19:18 Guest_84847 sleep is not doing Allah is doing
19:19 jesk I'am not sweating on the toilet without Allah doing
19:19 Guest_84847 hunger is not doing Allah is doing
19:19 post-factum Guest_84847: jerk off, we believe in Ganesha and St. Patrick here
19:19 Guest_84847 food does not take away the hunger Allah takes away the hunger
19:20 post-factum hagarth: just in time?
19:20 Guest_84847 water does not take away the thirst Allah takes away the thirst
19:20 Guest_84847 seeing is not doing Allah is doing
19:20 Guest_84847 hearing is not doing Allah is doing
19:20 Guest_84847 seasons are not doing Allah is doing
19:21 Guest_84847 weather is not doing Allah is doing
19:21 jesk you getting removed by Allah doing
19:21 Guest_84847 humans are not doing Allah is doing
19:21 Guest_84847 was kicked by hagarth: NO SPAMMING
19:21 post-factum Guest_84847: bye
19:21 post-factum hagarth++
19:21 glusterbot post-factum: hagarth's karma is now 7
19:22 jesk thank you Allah, for doing
19:22 johnmilton joined #gluster
19:23 post-factum it was Krishna, I guess
19:24 amye joined #gluster
19:27 shellclear joined #gluster
19:34 shellclear left #gluster
19:35 shellclear joined #gluster
19:37 shellclear hi everyone! how it works nfs access to a glusterfs?  I've mounted a glusterfs using nfs client but all the time when i tried create a file i'm receiving a access denied message
19:37 kovshenin joined #gluster
19:38 shellclear do I need to use posix acl?
19:39 JoeJulian shellclear: No, it's pretty standard stuff. Are you using gluster's nfs service? How are you mounting?
19:40 johnmilton joined #gluster
19:42 plarsen joined #gluster
19:42 shellclear JoeJulian: according to documentation, I'm using the following  command "sudo mount -t nfs -o vers=3,mountproto=tcp ipaddress:/exportpath /mountpoint"  at client side
19:43 JoeJulian And "exportpath" is your volume name?
19:43 shellclear JoeJulian: Yes
19:44 JoeJulian Do you have selinux enabled on your client or servers?
19:44 shellclear JoeJulian: I'm not using ganesha
19:53 JoeJulian Do you have selinux enabled on your client or servers?
19:55 shellclear Yes, I have.
19:55 msvbhat joined #gluster
19:56 JoeJulian Try disabling enforcement and see if that solves your problem. If it does, then you just have to figure out how to configure selinux.
19:56 Guest_84847 joined #gluster
19:57 shellclear JoeJulian: ok I'll try it
19:58 Guest_84847 left #gluster
19:59 shellclear JoeJulian: Do you think it's necessary to disable in all hosts?
19:59 JoeJulian You could probably find out by checking through your audit logs.
20:00 jdossey joined #gluster
20:02 shellclear JoeJulian: ok, thank you very much, I 'll try and afterwards I come here
20:02 shellclear left #gluster
20:02 JoeJulian Good luck
20:07 theron joined #gluster
20:24 sandersr joined #gluster
20:34 theron joined #gluster
20:34 shyam joined #gluster
20:55 congpine joined #gluster
20:56 theron joined #gluster
21:00 congpine hi all, can someone help with this simple question please? My replicated volume has a failed node and Im re-building it. However, I want to bring 1 brick online to re-sync only, not 8 of them. Is there any command to start glusterd with 1 brick ?
21:01 JoeJulian No. What I do is go ahead and do my "start...force" then I just kill the ones I don't want.
21:02 congpine thanks @JoeJulian. That's what I did, but when I start 8 bricks, it seems to fed up Gluster and I saw slow performance.
21:02 congpine Do you think having 8 bricks online and re-sync will affect Gluster performance ?
21:02 JoeJulian Yep
21:04 JoeJulian So, let's say you have /data/brick{1,2,3,4,5,6,7,8}. "gluster volume start $myvol force ; pkill -f brick[2-8]" will just leave the server for brick1 up, let it heal, then do the same with "[3-8]" etc.
21:08 congpine ah I see. my volume is distributed-replica so the volume is already up. But it still took me around 5 mins to kill the glusterd last time. and when I hit slow performance, most of the clients produced the following logs :
21:08 congpine [2016-08-10 21:07:18.301916] W [common-utils.c:2611:gf_process_reserved_ports] 5-glusterfs: Not able to get reserved ports, hence there is a possibility that glusterfs may consume reserved port
21:08 congpine [2016-08-10 21:07:15.297791] W [common-utils.c:2578:gf_get_reserved_ports] 5-glusterfs: could not open the file /proc/sys/net/ip v4/ip_local_reserved_ports for getting reserved ports info (No such file or directory)
21:09 congpine "/proc/sys/net/ip v4/ip_local_reserved_ports" isn't available on the clients or 4 x Gluster Servers
21:09 congpine do you have any ideas what was going wrong ?
21:10 JoeJulian That's just a warning. Gluster tries to honor any port reservations done through sys.
21:10 JoeJulian More than likely, it's all cpu and io, hashing the chunks of files that need healed.
21:11 JoeJulian Another possibility is to unmount the bricks you don't want to start. Start the remaining brick with "gluster volume start $myvol" (without the "force") and it won't start the bricks that are unmounted because they won't have the volume-id extended attribute.
21:12 JoeJulian And yes, I realize the volume is already started and running. That command doubles for starting the bricks that are down.
21:13 congpine yeah, that's the other way. It took around 5h for Gluster to really calm down and the read/write was worst . we are running 3.5.4-2 , when I checked heal info, it failed:
21:14 congpine gluster volume heal vol01 info
21:14 congpine Volume heal failed
21:15 congpine if it happens again, do you know if there is a way to troubleshoot or stop this heal process ?
21:23 JoeJulian hmm.. that old system is pretty much as-is at this point.
21:31 congpine @JoeJulian : do you have or know any resources to learn more about troubleshooting Gluster when it isn't performing as expected ? Logs sometime doesn't say much and i'd like to run commands to find out exactly what the problems are
21:37 JoeJulian congpine: state dumps are the next most useful information you can get, but they're extremely cryptic. Since you're not using a current version, you'll have to send SIGUSR1 to get the state dump. iirc, that version dumps it in /tmp.
21:38 congpine thanks JoeJulian. I will do more research about this.
21:38 congpine Have a good night
21:48 jdossey joined #gluster
21:54 theron joined #gluster
22:32 wadeholler joined #gluster
22:37 Wizek_ joined #gluster
22:47 hagarth joined #gluster
22:53 xoritior joined #gluster
22:53 xoritior hello
22:53 glusterbot xoritior: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
22:55 xoritior is mixing 3.7.13 and 3.8.0 reliable?  i tend to be wary of .0 releases
22:55 xoritior or should i stick with 3.7.13
22:55 xoritior it works fine
22:55 xoritior i have a node that puked the OS (the data is fine)
22:55 JoeJulian 3.8.2 is out
22:56 xoritior so i rebuilt and was about to rejoin...
22:56 xoritior oh dang
22:56 JoeJulian I'd stick with 3.7.13, at this point, unless you need features in 3.8.
22:56 xoritior i didnt see that... was looking in the repo
22:56 xoritior JoeJulian, is there a reason for that?
22:57 JoeJulian Mostly because I'm lazy.
22:57 xoritior or just a general keep it the same (kiss philosophy)
22:57 xoritior bwahahahaha
22:57 JoeJulian But if you're going to upgrade one, upgrade all.
22:57 xoritior you see that article about lazy people are smart people
22:57 xoritior righto
22:58 JoeJulian I haven't, but I sometimes call it efficient instead of lazy.
22:58 xoritior anything truely compelling in 3.8
22:58 * xoritior agrees with the efficient saying
23:03 xoritior JoeJulian, is there anything truely worth upgrading all my nodes to 3.8.2 for?
23:04 JoeJulian I forget which features are new in 3.8.
23:04 xoritior im looking for a changelog
23:05 xoritior https://www.gluster.org/community/roadmap/3.8/
23:05 glusterbot Title: Release Schedule Gluster (at www.gluster.org)
23:05 xoritior is that it?
23:09 xoritior hmm... i think i am too lazy to do this right now
23:16 JoeJulian :D
23:17 xoritior hmm... is there any way to force a peer to sync (rebuilt system same ip)
23:17 xoritior data is still there, but the nodes say it is already in the list
23:17 xoritior i think its a uuid mismatch
23:19 xoritior should i force a detach from a working node and probe the new one?
23:19 JoeJulian pull the uuid from an existing peer (/var/lib/glusterd/peers) and put it in /var/lib/glusterd/glusterd.info
23:20 xoritior well it looks like it may have fixed itself
23:20 xoritior no your right
23:20 xoritior i have State: Peer Rejected (Connected)
23:20 xoritior heh
23:22 xoritior State: Accepted peer request (Connected)
23:23 xoritior JoeJulian, if you are ever i mean EVER in Austin i owe you at least 1 beer
23:29 xoritior How do you drown a Hipster?
23:29 xoritior in the mainstream
23:32 JoeJulian xoritior: or if you come to Seattle... :D
23:32 xoritior ooooohhhh yea my wife wants to go up there for vacation some time
23:33 JoeJulian I usually avoid getting in the way of family vacations.
23:33 JoeJulian I wanted to hang out with Major when he was here, but I laid low so he could enjoy family time.
23:33 xoritior i understand that
23:34 xoritior i had to ditch one of my buddies up in nyc when i was there :-( not enough time
23:34 xoritior but then we were only there for 3 days
23:34 JoeJulian So what do you do in Austin?
23:35 xoritior i do IT for a few companies and some consulting...
23:35 wadeholler joined #gluster
23:36 xoritior mostly stuff that is fun
23:36 xoritior ;-)
23:38 xoritior other than one node trying to implode... its pretty fun
23:38 JoeJulian I was self-employed for 14 years. I loved it.
23:39 xoritior if it works it is great, it helps that my wife is a teacher and gets us good health insurance
23:39 JoeJulian Yeah, that was the only hard part.
23:39 xoritior yessir, that is a kick in the pants
23:40 JoeJulian I liked the freedom to be able to go to every one of my son's field trips when he was growing up.
23:41 xoritior yea, i get to take them to school, sometimes eat lunch with them, and usually do not have to miss any events
23:41 xoritior i say usually, because i have had to miss a few over the last couple of years
23:42 JoeJulian Well, true. Occasionally there were unforseen events.
23:43 xoritior my problem is knowing when to stop
23:43 xoritior i start working usually around 07:00 and have to force myself to stop
23:43 xoritior heh
23:44 xoritior but we took 3 vacations this summer and a couple of others during the year
23:44 masuberu joined #gluster
23:44 JoeJulian Yep, that's how I did it too.
23:45 JoeJulian Those midweek vacations are the best.
23:45 xoritior yea, airfare is dirt cheap and it just seems to work better

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary