Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-03-11

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 johnmilton joined #gluster
00:21 johnmilton joined #gluster
00:34 chirino joined #gluster
00:35 ovaistariq joined #gluster
00:36 cliluw joined #gluster
00:51 EinstCrazy joined #gluster
00:56 plarsen joined #gluster
01:02 haomaiwang joined #gluster
01:19 rwheeler joined #gluster
01:27 Lee1092 joined #gluster
01:54 ahino joined #gluster
02:04 harish_ joined #gluster
02:06 bit4man joined #gluster
02:24 baojg joined #gluster
02:27 haomaiwa_ joined #gluster
02:36 ovaistariq joined #gluster
02:41 lanning joined #gluster
02:54 lanning joined #gluster
03:01 haomaiwa_ joined #gluster
03:03 LessSeen_ joined #gluster
03:03 DV joined #gluster
03:08 haomaiwa_ joined #gluster
03:16 baojg joined #gluster
03:21 NIX0uT1aW_ joined #gluster
03:22 glisigno1i joined #gluster
03:29 harish joined #gluster
03:31 nthomas joined #gluster
03:35 haomaiwang joined #gluster
03:41 harish joined #gluster
03:42 hgowtham joined #gluster
03:44 Manikandan joined #gluster
03:45 atinm joined #gluster
03:45 overclk joined #gluster
03:46 nbalacha joined #gluster
03:58 shubhendu joined #gluster
03:58 harish joined #gluster
04:01 haomaiwa_ joined #gluster
04:03 RameshN_ joined #gluster
04:11 baojg joined #gluster
04:12 poornimag joined #gluster
04:17 ashiq joined #gluster
04:22 itisravi joined #gluster
04:29 Merlin_ joined #gluster
04:33 jkroon joined #gluster
04:38 poornimag joined #gluster
04:39 alghost joined #gluster
04:41 nehar joined #gluster
04:48 dthrvr joined #gluster
04:49 jiffin joined #gluster
04:54 baojg joined #gluster
04:54 skoduri joined #gluster
04:59 prasanth joined #gluster
05:01 haomaiwa_ joined #gluster
05:01 ppai joined #gluster
05:02 ndarshan joined #gluster
05:03 karthikfff joined #gluster
05:09 Merlin_ joined #gluster
05:11 calavera joined #gluster
05:13 nehar joined #gluster
05:15 Merlin_ joined #gluster
05:19 pur joined #gluster
05:21 aravindavk joined #gluster
05:26 Apeksha joined #gluster
05:27 nathwill joined #gluster
05:28 Bhaskarakiran joined #gluster
05:34 vmallika joined #gluster
05:37 chirino joined #gluster
05:41 kdhananjay joined #gluster
05:45 poornimag joined #gluster
05:48 btpier joined #gluster
05:49 bennyturns joined #gluster
05:49 wnlx joined #gluster
05:50 shyam joined #gluster
05:50 pur joined #gluster
05:50 ghenry joined #gluster
05:51 muneerse2 joined #gluster
05:51 nthomas joined #gluster
05:52 atinm joined #gluster
05:54 Wizek joined #gluster
05:56 sloop joined #gluster
05:57 Telsin joined #gluster
06:01 haomaiwang joined #gluster
06:02 bit4man joined #gluster
06:02 R0ok_ joined #gluster
06:02 d0nn1e joined #gluster
06:02 lh joined #gluster
06:02 tyler274 joined #gluster
06:02 k-ma joined #gluster
06:02 foster joined #gluster
06:02 atrius_ joined #gluster
06:02 purpleidea joined #gluster
06:02 ramky joined #gluster
06:03 karnan joined #gluster
06:04 RameshN_ joined #gluster
06:09 Merlin__ joined #gluster
06:12 primusinterpares joined #gluster
06:20 EinstCrazy joined #gluster
06:22 Merlin_ joined #gluster
06:23 Wizek joined #gluster
06:23 rafi joined #gluster
06:28 arcolife joined #gluster
06:34 Merlin_ joined #gluster
06:43 baojg joined #gluster
06:52 Merlin_ joined #gluster
06:57 baojg joined #gluster
06:59 mhulsman joined #gluster
07:01 64MAAJ2PB joined #gluster
07:01 DV joined #gluster
07:05 Merlin_ joined #gluster
07:06 nehar joined #gluster
07:08 ayma joined #gluster
07:15 DV__ joined #gluster
07:15 Merlin_ joined #gluster
07:21 ayma joined #gluster
07:21 R0ok_ joined #gluster
07:21 d0nn1e joined #gluster
07:21 lh joined #gluster
07:21 tyler274 joined #gluster
07:21 k-ma joined #gluster
07:21 foster joined #gluster
07:21 atrius_ joined #gluster
07:21 purpleidea joined #gluster
07:26 Merlin_ joined #gluster
07:29 harish_ joined #gluster
07:33 DaKnOb joined #gluster
07:36 Merlin_ joined #gluster
07:37 armyriad joined #gluster
07:50 d0nn1e joined #gluster
07:54 ayma joined #gluster
07:54 R0ok_ joined #gluster
07:54 lh joined #gluster
07:54 tyler274 joined #gluster
07:54 k-ma joined #gluster
07:54 foster joined #gluster
07:54 atrius_ joined #gluster
07:54 purpleidea joined #gluster
07:55 Merlin_ joined #gluster
07:56 ayma left #gluster
07:58 deniszh joined #gluster
08:01 harish_ joined #gluster
08:01 18VAAESCS joined #gluster
08:03 DV joined #gluster
08:07 Merlin_ joined #gluster
08:09 unlaudable joined #gluster
08:14 ayma1 joined #gluster
08:14 ayma1 left #gluster
08:17 Merlin_ joined #gluster
08:20 ctria joined #gluster
08:28 Merlin_ joined #gluster
08:31 deniszh joined #gluster
08:32 gem joined #gluster
08:34 jri joined #gluster
08:42 Merlin__ joined #gluster
08:42 DaKnOb joined #gluster
08:48 unlaudable joined #gluster
08:51 Slashman joined #gluster
08:53 Merlin_ joined #gluster
08:56 [Enrico] joined #gluster
08:58 ayma joined #gluster
08:59 ayma1 joined #gluster
09:00 itisravi joined #gluster
09:01 haomaiwa_ joined #gluster
09:01 kotreshhr joined #gluster
09:02 LessSeen_ joined #gluster
09:03 Saravanakmr joined #gluster
09:04 DV joined #gluster
09:05 hchiramm joined #gluster
09:06 jri joined #gluster
09:07 Merlin_ joined #gluster
09:10 spalai joined #gluster
09:18 Merlin_ joined #gluster
09:20 anil joined #gluster
09:24 sagarhani joined #gluster
09:26 GreatSnoopy joined #gluster
09:32 spalai joined #gluster
09:37 nix0ut1aw joined #gluster
09:40 Wizek joined #gluster
09:41 Merlin_ joined #gluster
09:42 jww Hello.
09:42 glusterbot jww: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:47 rafi jww: hello
09:47 jakob___1 I have a _VERY_ weird behaviour with a simple glustre (3.7.5) setup. It's just two replicated bricks (on raid) with 40Gbps ethernet. Two running fio seems to work ok, but with four clients I get _REPRODUCABLE_ very different results, depening on which directory I put the benchmark (fio) files. Aggregated BW from all clients can go from 50MB/s to 900MB/s ?! This is achived with _identical_ parameter, only thing that differs is the directory name... Anyo
09:49 jakob___1 I know gluster uses hashing on filenames (what about directories??) to distribute load, but this is a setup with just two storage servers and one brick each (replicated)....I can not understand how hashing could affect server load..
09:51 jww I'm benchmarking a gluster share with iozone / smallfile but results are low. for example on file creation I get 5.2MB/sec for a 1gb file with iozone and 10 000 64k files, I get 0.84MB/sec . any idea on why it's so slow ? there are 2 nodes with 1GB network.
09:53 deniszh joined #gluster
09:54 rafi jww: I don't have a result for small file benchmarking
09:55 rafi jww: may be bennyturns can help you
09:57 jww rafi: it's slow even for large files ;)
09:57 jakob___1 jww: Writing a large 1GB file is just 5.2MB/sec? How many parallel?
09:57 jww 8
09:58 jww I mean 8 threads to write this 1GB
09:59 jakob___1 What hardware? Is 5.2MB/s total aggregated BW or is it per thread BW? Seems low anyway...unless you run on some antique hw.
10:00 Merlin_ joined #gluster
10:01 haomaiwa_ joined #gluster
10:01 jww it's some intel E5 cpus with software RAID 1 and  32Gb of ram. 5.2MB/sec  is the total, the exact value (from iozone ) is 42183.17 KB/sec
10:09 jakob___1 jww: If that sequential io w. iozone, then there is an issue even with large files. 5.2MB/s is very low...
10:10 Manikandan joined #gluster
10:10 jakob___1 What version are you using? I know a serious(?) bug appeared in 3.7.7 that affected write performance.
10:13 deniszh1 joined #gluster
10:15 jww jakob___1: it's version 3.2
10:19 Marbug could it be good to share some logs on a glusterfs partition? like apache log files
10:19 post-factum jww: omg, you'd better update
10:20 jww post-factum: in fact I'm here to upgrade, or to change gluster. but before I need to know why it's slow.
10:21 post-factum jww: network storage slowness is because of 1) latency; 2) replica misconfiguration
10:21 DV joined #gluster
10:22 post-factum jww: you'd really need to consider your real workload, upgrade and tune cluster according to workload
10:22 Merlin_ joined #gluster
10:24 jww post-factum: I think network is ok( it's ovh's network ) I don't know about replica misconfiguration , but since there is only 2 nodes the setup is pretty simple.
10:25 jww post-factum: how do you suggest to check my real workload ? up to now I just checked the average size of the files and did read/write operations.
10:26 ivan_rossi joined #gluster
10:27 spalai joined #gluster
10:27 post-factum 1) what is your *real* workload?
10:27 post-factum 2) what is RTT between nodes?
10:27 post-factum 3) what is your volume configuration?
10:32 shubhendu joined #gluster
10:32 jww 1) I don't know how to know my real workload. there is no graph or monitoring on this serveur.
10:33 jww 2) RTT is around 0.075ms
10:36 jww 3) here is my volume configuration : http://dpaste.com/11ZWBJ7
10:36 glusterbot Title: dpaste: 11ZWBJ7 (at dpaste.com)
10:36 post-factum 1) what is being stored on gluster, what apps work with it etc etc etc
10:37 RayTrace_ joined #gluster
10:37 Iouns joined #gluster
10:38 jww post-factum: files ( mainly pdf ) from 200mb to 2GB , the average size in 1GB . the apps working with it a are custom java program .also the mount is using glusterfs and not nfs
10:40 jakob___1 Anyone who has some idea why different directory names(!!?) causes a massive performance impact (details here in irc, about an hour ago)? post-factum?
10:40 Merlin_ joined #gluster
10:42 post-factum jww: do those apps produce massive I/O?
10:44 jww post-factum: not that much
10:44 post-factum jww: so, just update and do not pay an attention iozone results
10:44 post-factum i guess they are highly irrelevant for network fs
10:45 jww post-factum: umm, I guess I should not ignore users that tell me it's very slow.
10:45 jww because I'm not able to setup/debug gluster, I'll have to install another network fs instead of upgrading.
10:46 gem joined #gluster
10:47 RayTrace_ joined #gluster
10:47 post-factum jww: it is up to you, but using old untuned version and claiming it doesn't work properly is not the most wise thing i've ever seen
10:48 jww post-factum: yes you are right.
10:48 DaKnOb joined #gluster
10:54 DaKnOb joined #gluster
10:54 jww thanks for your answers, I'll upgrade and come back next week !
11:00 Merlin_ joined #gluster
11:01 haomaiwa_ joined #gluster
11:05 gem joined #gluster
11:11 RayTrace_ joined #gluster
11:13 haomaiwang joined #gluster
11:13 ira joined #gluster
11:14 post-factum anyone knows, how to cope with "Stale file handle" error on glusterfs fuse mount?
11:15 jri joined #gluster
11:15 pgreg joined #gluster
11:16 ndarshan joined #gluster
11:17 haomaiwa_ joined #gluster
11:18 skoduri post-factum, probably re-mount the share?
11:19 post-factum skoduri: didn't help. i went through all bricks (it is distrubuted volume) and removed file manually from some of them. then it disappeared in mount
11:19 post-factum skoduri: it is .git/config file created by git
11:19 post-factum skoduri: and it is 3.7.6
11:19 post-factum skoduri: log says that LOOKUP() returns -1
11:20 RameshN joined #gluster
11:21 post-factum skoduri: it happened twice to me, both times, i guess, it was file creation
11:21 deniszh joined #gluster
11:24 post-factum skoduri: i will try to create test volume and run massive git init/add/commit to simulate the load
11:27 Merlin_ joined #gluster
11:34 unlaudable joined #gluster
11:36 jakobs I'm trying to debug my issue posted below and I now notice that it seems that seq write with a simple dd (using odirect) actually performs better than fio with async io..... I was expecting the oposite... Any ideas why fio is not performing as well with multiple clients?
11:36 skoduri post-factum, okay
11:37 ira_ joined #gluster
11:37 jakobs I found a parameter called "storage.linux-aio" that is off (by default), could it be related to clients running async io?
11:38 skoduri post-factum, you can try collecting pkt trace on the server and check if the brick process is returning error for lookup...
11:38 skoduri post-factum, and if that is the case , we need to check why the lookup failed when the file is actually present at the backend
11:39 deniszh joined #gluster
11:41 robb_nl joined #gluster
11:44 RayTrace_ joined #gluster
11:48 EinstCrazy joined #gluster
11:52 DV__ joined #gluster
11:53 post-factum skoduri: yup, sure. just let me deploy test setup now
11:53 skoduri post-factum, sure
11:54 RameshN joined #gluster
11:54 post-factum skoduri: the only thing i've already noticed is that the file was actually present on two of five bricks
11:54 post-factum skoduri: on one brick it was zero file
11:54 post-factum skoduri: on another it was the actual content
11:54 post-factum skoduri: and no files on other three bricks
11:54 skoduri post-factum, are those two bricks part of replica-pair?
11:54 R0ok_ joined #gluster
11:55 post-factum skoduri: all 5 form one distributed volume
11:55 post-factum pure distributed
11:55 post-factum no replica
11:56 Merlin_ joined #gluster
11:56 Merlin_ joined #gluster
11:57 skoduri okay.... spalai  ^^^.. any idea?
11:57 Merlin_ joined #gluster
11:57 hackman joined #gluster
11:58 post-factum skoduri: just tell me now if there was some fixes regarding this after 3.7.6
11:58 post-factum skoduri: if you could remember of course
12:00 johnmilton joined #gluster
12:04 Gnomethrower joined #gluster
12:12 skoduri post-factum, sorry I am not sure about dht fixes..
12:15 spalai joined #gluster
12:15 deniszh joined #gluster
12:22 rafi joined #gluster
12:22 spalai left #gluster
12:23 post-factum skoduri: probably, this is related: https://bugzilla.redhat.co​m/show_bug.cgi?id=1283036
12:23 glusterbot Bug 1283036: unspecified, unspecified, ---, aspandey, POST , Index entries are not being purged in case of file does not exist
12:23 post-factum skoduri: it is fixed, however, after 3.7.6
12:23 andy-b joined #gluster
12:25 DaKnOb joined #gluster
12:26 Wizek joined #gluster
12:28 post-factum skoduri: also, get the following error in several bricks logs: http://termbin.com/w05a
12:32 post-factum skoduri: another observation. i'm running git test workload now and get the same "File exists" error on .git/config. however, no stalled handles so far
12:33 post-factum skoduri: it occurs in posix_handle_hard and posix_mknod as well
12:34 post-factum skoduri: also, the following error appears: http://termbin.com/i8zh
12:34 post-factum skoduri: sorry for spam :)
12:39 anil joined #gluster
12:41 skoduri post-factum, the bug is filed against replicate/afr but since yours is pure distributed volume, it may not be the issue
12:42 Merlin_ joined #gluster
12:45 ndarshan joined #gluster
12:46 skoduri post-factum, you could post your findings in gluster-users...someone more familiar with dht shall respond hopefully :)
12:47 post-factum skoduri: will re-test with 3.7.8 now and surely post to ML
12:50 baojg joined #gluster
12:56 Merlin_ joined #gluster
12:57 post-factum skoduri: cannot reproduce with 3.7.8. seems to be fixed
12:58 Wizek joined #gluster
13:06 anti[Enrico] joined #gluster
13:07 andy-b joined #gluster
13:08 gem joined #gluster
13:14 bit4man joined #gluster
13:15 sebamontini joined #gluster
13:18 anti[Enrico] joined #gluster
13:19 spalai joined #gluster
13:20 Merlin_ joined #gluster
13:23 DaKnOb joined #gluster
13:32 plarsen joined #gluster
13:39 ninkotech joined #gluster
13:39 ninkotech_ joined #gluster
13:41 chirino joined #gluster
13:42 shubhendu joined #gluster
13:44 kanagaraj joined #gluster
13:44 B21956 joined #gluster
13:49 jri joined #gluster
13:51 shaunm joined #gluster
13:53 armyriad joined #gluster
13:53 rafi joined #gluster
13:57 spalai left #gluster
13:57 ndarshan joined #gluster
14:00 nehar joined #gluster
14:05 Hamburglr joined #gluster
14:07 Hamburglr the heal info split-brain command seems to just be hanging, is there any way to check if it's actually doing anything?
14:09 Hamburglr or cancel it safely?
14:12 kanagaraj joined #gluster
14:15 Merlin_ joined #gluster
14:18 unclemarc joined #gluster
14:18 kanagaraj_ joined #gluster
14:19 clyons joined #gluster
14:27 aravindavk joined #gluster
14:28 Wizek joined #gluster
14:31 kanagaraj__ joined #gluster
14:31 DV joined #gluster
14:33 haomaiwa_ joined #gluster
14:39 Merlin_ joined #gluster
14:40 DV joined #gluster
14:40 nbalacha joined #gluster
14:44 Merlin_ joined #gluster
14:51 mpietersen joined #gluster
14:52 Merlin_ joined #gluster
14:53 mpietersen joined #gluster
14:55 kotreshhr left #gluster
15:01 haomaiwa_ joined #gluster
15:05 ahino joined #gluster
15:25 coredump joined #gluster
15:27 Merlin_ joined #gluster
15:27 amye joined #gluster
15:28 bennyturns joined #gluster
15:39 hagarth joined #gluster
15:42 ahino joined #gluster
15:43 raghu joined #gluster
15:50 farhorizon joined #gluster
15:50 ayma joined #gluster
15:54 Merlin_ joined #gluster
16:00 wushudoin joined #gluster
16:01 haomaiwang joined #gluster
16:05 spalai joined #gluster
16:05 Merlin_ joined #gluster
16:09 wushudoin joined #gluster
16:11 robb_nl joined #gluster
16:15 hagarth joined #gluster
16:18 skylar joined #gluster
16:24 haomaiw__ joined #gluster
16:25 ninjaryan joined #gluster
16:27 farhoriz_ joined #gluster
16:32 RameshN joined #gluster
16:38 Merlin_ joined #gluster
16:38 hchiramm joined #gluster
16:48 shaunm joined #gluster
16:48 Manikandan joined #gluster
16:51 skylar joined #gluster
16:53 dblack joined #gluster
17:01 haomaiwa_ joined #gluster
17:06 Slashman joined #gluster
17:09 skylar joined #gluster
17:20 calavera joined #gluster
17:26 farhoriz_ I am performance tuning & the gluster volume profile stats are dominated by stat calls.   I tried to turn off nfs prefetch, but I don't see any difference.   Is gluster volume-set <brick> performance.nfs.stat-prefetch off the right thing to set still?
17:30 farhoriz_ eh, sorry, not volume-set ... volume set.   It accepted the command without error, I just don't see any change in behavior
17:32 post-factum farhoriz_: have you restarted/remounted the volume?
17:32 farhoriz_ I have not.  Should I remount or should I restart?
17:32 spalai joined #gluster
17:34 pur joined #gluster
17:37 post-factum both
17:38 farhoriz_ @post-factum thanks, retrying with that
17:44 Merlin_ joined #gluster
17:56 Merlin_ joined #gluster
17:58 amye joined #gluster
18:01 sebamontini joined #gluster
18:01 haomaiwa_ joined #gluster
18:03 shubhendu joined #gluster
18:05 Merlin_ joined #gluster
18:05 ivan_rossi left #gluster
18:05 shubhendu joined #gluster
18:09 farhoriz_ @post-factum hmm, still seeing tons more stat calls than file access/etc.   Like 1.1 M stats vs 18 K file access.   Going to try restarting gluster services as well next
18:11 skylar joined #gluster
18:22 sebamontini hi, any best practices to setup a 2 node gluster in EC2 ? we're having the VMs with a really high Loadavg
18:26 wushudoin joined #gluster
18:28 calavera joined #gluster
18:37 Merlin_ joined #gluster
18:37 ahino joined #gluster
18:49 wushudoin joined #gluster
18:49 wushudoin joined #gluster
19:01 haomaiwa_ joined #gluster
19:04 skylar joined #gluster
19:13 Merlin_ joined #gluster
19:14 jiffin joined #gluster
19:19 hackman joined #gluster
19:20 spalai left #gluster
19:23 calavera joined #gluster
19:25 sebamontini joined #gluster
19:32 ovaistariq joined #gluster
19:35 Merlin_ joined #gluster
19:36 ovaistar_ joined #gluster
19:37 gbox joined #gluster
19:44 sebamontini joined #gluster
19:49 Merlin_ joined #gluster
19:57 chirino_m joined #gluster
19:59 petan joined #gluster
20:01 haomaiwa_ joined #gluster
20:19 R0ok_ joined #gluster
20:26 dblack joined #gluster
20:26 DV joined #gluster
20:26 Merlin_ joined #gluster
20:29 ovaistariq joined #gluster
20:31 calavera joined #gluster
20:32 LessSeen_ joined #gluster
20:33 unlaudable joined #gluster
20:42 hagarth joined #gluster
20:47 sebamontini joined #gluster
20:51 d0nn1e joined #gluster
20:52 Merlin_ joined #gluster
21:01 haomaiwa_ joined #gluster
21:08 deniszh joined #gluster
21:13 calavera joined #gluster
21:15 Wizek joined #gluster
21:18 mhulsman joined #gluster
21:27 skylar joined #gluster
21:27 ovaistariq joined #gluster
21:30 ovaistar_ joined #gluster
21:33 bennyturns joined #gluster
21:35 Merlin_ joined #gluster
21:57 Merlin_ joined #gluster
21:58 csterling joined #gluster
21:58 ctria joined #gluster
21:58 csterling Hey guys!
22:00 csterling I have a new question - right after I create a new ext4 partition and mount it for glusterfs, how can I test that I have the format correct before I try to add it to the rest of the cluster
22:00 csterling I was trying something like this
22:00 csterling mount -t glusterfs gluster01.newmediadenver.com:/srv/sdj1 /mnt/gluster/
22:00 csterling or just mount -t glusterfs /srv/sdj1 /mnt/gluster/
22:00 csterling But I get a failed to get the 'volume file' from server
22:00 csterling error in the log
22:01 haomaiwa_ joined #gluster
22:01 csterling Is it just that I need to add it to an actual gluster volume before I can mount it?
22:02 csterling I thought I needed to mount it to add the brick/volume name in a directory, but I feel like I’m doing something wrong
22:02 m0zes yes, it needs to be part of a gluster volume before gluster knows about it.
22:02 csterling I kinda figured - I just wanted to make sure I was on the right track
22:08 DV joined #gluster
22:08 Merlin_ joined #gluster
22:09 ovaistariq joined #gluster
22:10 ovaistariq joined #gluster
22:15 skylar1 joined #gluster
22:18 csterling I’m currently trying to take a brick out of rotation, grow the volume, and bring it back into rotation. I see a lot of documentation about removing a bad brick - is that the documentation I should follow?
22:19 dlambrig_ joined #gluster
22:23 Merlin_ joined #gluster
22:25 csterling [root@gluster01 extras]# gluster volume rebalance nmd start
22:25 csterling volume rebalance: nmd: success: Rebalance on nmd has been started successfully. Use rebalance status command to check status of the rebalance process.
22:25 csterling ID: 2d879973-6299-4f82-a273-3c5044ba2e4c
22:25 csterling [root@gluster01 extras]# gluster volume rebalance nmd status
22:25 csterling volume rebalance: nmd: failed: Rebalance not started.
22:25 csterling Any idea why a rebalance would “start” but fail to start?
22:26 Wizek joined #gluster
22:27 dlambrig_ joined #gluster
22:29 post-factum csterling: logs?
22:29 csterling Which ones would you like?
22:30 csterling the rebalance log?
22:31 csterling http://pastebin.com/AGfPpdrd
22:31 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
22:31 csterling gotcha
22:32 post-factum yup, that one
22:32 post-factum "Found critical error from gf_defrag_get_entry", and nothing about *what* error :)
22:32 csterling Isnt’ that nice?
22:32 csterling I was running a rebalance
22:32 csterling And I stopped it
22:32 csterling I waited for it to finish
22:33 post-factum it seems it really rebalanced almost all files except 5
22:33 csterling So nothing to woyy about?
22:33 csterling Is there a good way to run a check?
22:34 post-factum in fact, "rebalance status" should show you something
22:34 post-factum not "failed" message
22:35 csterling [root@gluster01 glusterfs]# gluster volume rebalance nmd status
22:35 csterling volume rebalance: nmd: failed: Rebalance not started.
22:35 csterling right
22:35 csterling which says something is wrong :-(
22:35 post-factum anything in /var/lib/glusterd/glustershd/run?
22:35 csterling glustershd.pid
22:36 post-factum and /var/lib/glusterd/vols/nmd/rebalance
22:36 post-factum ?
22:36 csterling nada
22:37 post-factum /var/lib/glusterd/vols/nmd/nmd-rebalancevol
22:37 post-factum *rebalance.vol
22:38 csterling
22:38 csterling /var/lib/glusterd/vols/nmd/nmd-rebalance.vol is a file
22:38 post-factum yup, it must be :)
22:38 post-factum smth in it?
22:39 csterling http://ur1.ca/omlf7
22:39 glusterbot Title: #337414 Fedora Project Pastebin (at ur1.ca)
22:40 post-factum and what if you restart glusterd, then restart volume, then start rebalance again?
22:40 Merlin_ joined #gluster
22:41 csterling Restarting all of glusterd is unfortunately not an option
22:41 csterling production box
22:41 csterling I don’t think
22:41 post-factum meh
22:41 csterling If I just kill it on a single box, it should be OK?
22:42 post-factum ok, what if just start rebalance again?
22:42 post-factum it won't be ok, your clients will stuck for 42 secs
22:42 csterling [root@gluster01 nmd]# gluster volume rebalance nmd start
22:42 csterling volume rebalance: nmd: success: Rebalance on nmd has been started successfully. Use rebalance status command to check status of the rebalance process.
22:42 csterling ID: c75cc3da-f6c4-48c0-8bd3-d13e1d82ac2a
22:42 post-factum and status
22:42 csterling [root@gluster01 nmd]# gluster volume rebalance nmd status
22:42 csterling volume rebalance: nmd: failed: Rebalance not started.
22:42 post-factum i'd restart gust glusterd on one box
22:43 post-factum afaik, it is safe to do that unless you kill bricks as well
22:43 post-factum *just
22:43 csterling service flusterd restart
22:43 csterling And then anything else to get it working?
22:43 robb_nl joined #gluster
22:43 post-factum nope, just restart
22:43 csterling glusterd*
22:43 csterling cool
22:44 csterling Restarted
22:44 csterling try the rebalance
22:44 csterling ?
22:45 post-factum ye
22:45 csterling [root@gluster01 nmd]# gluster volume rebalance nmd start
22:45 csterling volume rebalance: nmd: success: Rebalance on nmd has been started successfully. Use rebalance status command to check status of the rebalance process.
22:45 csterling ID: 4822475c-ef79-49e3-a616-44ea7e598ec8
22:45 csterling [root@gluster01 nmd]# gluster volume rebalance nmd status
22:45 csterling volume rebalance: nmd: failed: Rebalance not started.
22:45 csterling same
22:45 csterling ps aux | grep gluster
22:45 csterling oops
22:45 post-factum try to inspect logs for some warning/error at current timestamp
22:46 csterling kk
22:46 csterling nothing in the rebalance log
22:47 post-factum glusterd log
22:47 csterling I have glustershd.log
22:47 csterling would that work?
22:48 post-factum i don't think so
22:48 csterling hrms
22:48 post-factum etc-glusterfs-glusterd.vol.log
22:49 csterling http://ur1.ca/omlgd
22:49 glusterbot Title: #337422 Fedora Project Pastebin (at ur1.ca)
22:49 hijakk_1 joined #gluster
22:49 Merlin_ joined #gluster
22:50 post-factum csterling: you could try to run glusterd in debug mode to get more info
22:50 csterling The creation of the listener?
22:50 hijakk_1 Howdy all, I posted a question to the list serv a while back but never got a response - was hoping perhaps I could get some ideas here
22:50 csterling Do you have some info on that? I’m assuming I change a setting somewhere
22:51 post-factum glusterd --debug
22:51 hijakk_1 Basically I want to be able to serve lots (millions+) of images via http. I'm hoping gluster could be a good way for me to get redundancy and scalability out of this
22:51 post-factum csterling: you could inspect /usr/lib/systemd/system/glusterd.service for that
22:51 csterling kk
22:52 hijakk_1 What should I watch out for when setting something like this up? Can I just point Apache at a mounted gluster volume and have at it?
22:52 post-factum hijakk_1: you have to consider 2 things
22:52 post-factum hijakk_1: 1) fuse mount for serving millions of files is a bit slow. you must stick to glusterfs api
22:53 post-factum hijakk_1: 2) you must carefully consider file/folder layout (hierarchy) for the storage. iow, you cannot keep 1M of files in one folder
22:53 csterling post-factum: http://ur1.ca/omlgp is the result of the command - the file you specified doesn’t seem to exist
22:53 hijakk_1 Great, ok. Yeah, I've got a structure where the first characters of a file's name route it in a directory hierarchy
22:53 glusterbot Title: #337425 Fedora Project Pastebin (at ur1.ca)
22:54 post-factum csterling: centos7?
22:54 shaunm joined #gluster
22:55 post-factum hijakk_1: you may stick to hashes as well. like /folder/ab/cd/0e/abcd0edeadbeef.file
22:55 post-factum hijakk_1: and store hash-file relation somewhere in DB
22:55 csterling 6.7
22:55 post-factum csterling: oh. then it is under /etc/init.d :)
22:55 csterling ahhh
22:55 hijakk_1 Yes indeed - the files are getting named with a sha1 hash actually.
22:56 post-factum hijakk_1: so you are ok with that
22:56 csterling ahh
22:56 hijakk_1 I know in raw volumes I can end up encountering inode problems with lots of small files - do I need to keep the same kind of eye on gluster?
22:57 post-factum csterling: probably, you have /etc/sysconfig/glusterd, and you may set loglevel there
22:57 csterling Yup - just found him
22:57 post-factum hijakk_1: just make sure you have enough of them
22:57 csterling loglevel=DEBUG
22:57 csterling and just specify a lofile?
22:58 post-factum csterling: just modify loglevel and start glusterd as usual. i guess
22:59 csterling restarting
22:59 nathwill joined #gluster
23:00 post-factum hijakk_1: what we have done: having millions of video chunks, we serve them to webserver via small app that proxies http GET calls to glusterfs api calls
23:00 post-factum hijakk_1: it is pretty easy to do
23:01 hijakk_1 Ok, cool - I'm assuming it'd be a custom app to access gluster, then, and not a custom app to expose gluster to something like apache?
23:01 haomaiwa_ joined #gluster
23:02 csterling post-factum: I’m not seeing any additional log file - let me try adding a file directive and see if that works
23:02 hijakk_1 http://www.gluster.org/community/documentat​ion/index.php/Running_Apache_with_Glusterfs looks potentially useful - the "booster" option
23:03 csterling post-factum: I’m looking around, but I’m not seeing any clues
23:03 post-factum csterling: you won't see additional files
23:03 post-factum csterling: you must see more info in existing files
23:03 csterling gotcha - it’s just super logging to all
23:03 csterling yup yup
23:04 csterling hmm
23:04 post-factum hijakk_1: i believe that doc is obsoleted
23:05 post-factum hijakk_1: long time ago
23:05 hijakk_1 darn :(
23:06 post-factum hijakk_1: https://github.com/davidjericho/mod_proxy_gluster
23:06 glusterbot Title: GitHub - davidjericho/mod_proxy_gluster: Modified mod_proxy_gluster supporting large files, ISO8601 times, and icons in much the same manner as the normal mod_autoindex from Apache (at github.com)
23:06 shyam left #gluster
23:06 post-factum hijakk_1: seems what you want
23:06 hijakk_1 fantastic, thanks post-factum
23:06 post-factum hijakk_1: never tried, though. try and share the experience :)
23:06 hijakk_1 will do
23:06 hijakk_1 Time to head home for the day!
23:07 post-factum hijakk_1: good luck/have fun
23:07 csterling post-factum: I’m digging through the same files, but I don’t see any additional info
23:07 csterling Here was the order of operations
23:07 csterling We added 2 new bricks to our distributed replicated file system
23:07 csterling Ran a gluster volume rebalance start
23:08 csterling Stopped that
23:08 csterling Ran a script Joe recommended https://github.com/gluster/gluster​fs/blob/master/extras/rebalance.py
23:08 glusterbot Title: glusterfs/rebalance.py at master · gluster/glusterfs · GitHub (at github.com)
23:08 csterling Set the xattrs off of that
23:09 csterling fattrs
23:09 csterling eg setfattr -n trusted.glusterfs.size-weighted -v 0x000000020000000000000000ec4ec4eb /tmp/tmpRAA0CL/brick1
23:09 csterling then tried to start the rebalance
23:09 csterling And that was when it stopped liking me
23:10 post-factum csterling: shy would you use some hand work to make the rebalance?
23:10 post-factum *why
23:10 post-factum why not just add bricks and start rebalance :)
23:10 csterling I added bricks and started the rebalance, but it started filling up our smaller bricks
23:11 post-factum ah, but i thought gluster consider inequivalent brick sizing
23:11 csterling I thought so too, but Joe said this script would help
23:11 post-factum it must fill bricks proportionally
23:11 csterling Is there a way to “reset” it?
23:11 post-factum umm, we need to cast Joe
23:11 post-factum JoeJulian: ^^
23:11 post-factum no answer from me on that :(
23:12 csterling JoeJulian: are you around?
23:12 post-factum csterling: in fact, it is friday-saturday night, and even me drinking wine should not be here :)
23:12 csterling I understand
23:13 post-factum at least he is online and will not miss the message
23:14 csterling post-factum: in your opinion, will this state cause an outage this weekend?
23:15 csterling The files still seem to be reading/writing
23:15 csterling (From a downstream view)
23:15 csterling And I’m just trying to figure out if I can pick this up Monday
23:16 post-factum i believe it won't cause outage
23:16 post-factum rebalance is just about force moving files around according to new bricks layout
23:16 post-factum new files will be placed according to new layout afaik
23:17 post-factum just make sure you have enough space on your bricks to survive the weekend
23:18 csterling Out of curiosity, if I have 6GB left on one brick, and 500GB left on another, will gluster failover to another brick when it has a cache-miss?
23:18 csterling Or do i just have 6GB left?
23:19 post-factum not sure how gluster behaves in caseof ENOSPACE
23:19 post-factum never faced that
23:19 csterling I wasn’t sure either, but the new bricks are completely open
23:19 csterling So hopefully well
23:21 csterling post-factum++
23:21 glusterbot csterling: post-factum's karma is now 4
23:21 csterling Thank you very much for working with me
23:24 swebb joined #gluster
23:32 JoeJulian Now wait a minute... I said to look at that script, not use it. :P
23:33 post-factum mwhahahah
23:33 skylar joined #gluster
23:33 JoeJulian If a brick exceeds the minimum free space, a file creation will move to the next dht subvolume. If, however, you add data to a preexisting file, that will continue to fill up the drive.
23:34 post-factum i know it should finish like oscar drama
23:35 post-factum JoeJulian: is brick selection algorithm proportionally-free-space-wise?
23:35 JoeJulian Nope
23:35 JoeJulian @lucky dht misses are expensive
23:35 glusterbot JoeJulian: https://joejulian.name/blog​/dht-misses-are-expensive/
23:35 JoeJulian The algorithm is explained there.
23:36 ira joined #gluster
23:36 post-factum that is why we need rebalance
23:37 csterling Right
23:37 post-factum i'm aware of range hashes
23:38 csterling That helps - new files get distributed, old ones grow - which should be fine in our cluster for the weekend
23:38 post-factum JoeJulian: one more question. if changing volume options, e.g., some performance ones, how it impacts on clients currently have the volume mounted? should we 1) restart the volume; 2) remount the volume; 3) both?
23:39 post-factum csterling: correct
23:42 Merlin_ joined #gluster
23:42 amye joined #gluster
23:45 samsaffron___ joined #gluster
23:46 unforgiven512 joined #gluster
23:46 unforgiven512 joined #gluster
23:50 billputer joined #gluster
23:56 Merlin_ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary