Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2015-07-16

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 haomaiwang joined #gluster
00:15 craigcabrey joined #gluster
00:21 calavera joined #gluster
00:29 craigcabrey joined #gluster
00:37 topshare joined #gluster
01:10 PatNarciso dijuremo, Romeor, _maserati_, doctorray: I too have horribly slow 'ls' times on a single-node, multi-brick xfs backed gluster volume.  not sure whats up.  setup was fine for about a month, then started getting really slow.
01:10 PatNarciso samba clients sometimes wait beyond 30-90s for a dir list of 100 files.  meanwhile an 'ls' on the xfs mount itself is swift.  144k files; 32TB of 44TB used; everything from video to .DS_Store.
01:23 atrius joined #gluster
01:23 harish joined #gluster
01:23 topshare joined #gluster
01:24 Lee1092 joined #gluster
01:32 PatNarciso joined #gluster
01:39 lpabon joined #gluster
01:47 ilbot3 joined #gluster
01:47 Topic for #gluster is now Gluster Community - http://gluster.org | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - https://botbot.me/freenode/gluster/ & http://irclog.perlgeek.de/gluster/
01:52 craigcabrey joined #gluster
01:53 edwardm61 joined #gluster
02:06 kdhananjay joined #gluster
02:11 Peppaq joined #gluster
02:29 RedW joined #gluster
02:41 calisto joined #gluster
02:51 nangthang joined #gluster
02:52 julim joined #gluster
02:53 nangthang joined #gluster
02:56 glusterbot News from newglusterbugs: [Bug 1243642] GF_CONTENT_KEY should not be handled unless we are sure no other operations are in progress <https://bugzilla.redhat.com/show_bug.cgi?id=1243642>
02:56 glusterbot News from newglusterbugs: [Bug 1243644] Metadata self-heal is not handling failures while heal properly <https://bugzilla.redhat.com/show_bug.cgi?id=1243644>
02:56 glusterbot News from newglusterbugs: [Bug 1243647] Disperse volume : data corruption with appending writes in 8+4 config <https://bugzilla.redhat.com/show_bug.cgi?id=1243647>
02:56 glusterbot News from newglusterbugs: [Bug 1243648] Disperse volume: NFS crashed <https://bugzilla.redhat.com/show_bug.cgi?id=1243648>
03:07 lpabon joined #gluster
03:09 dijuremo @PatNarciso I feel your pain... I am not sure what else to try... :(
03:15 vmallika joined #gluster
03:21 TheSeven joined #gluster
03:26 glusterbot News from newglusterbugs: [Bug 1243654] fops fail with EIO on nfs mount after add-brick and rebalance <https://bugzilla.redhat.com/show_bug.cgi?id=1243654>
03:26 glusterbot News from newglusterbugs: [Bug 1243655] Sharding - Use (f)xattrop (as opposed to (f)setxattr) to update shard size and block count <https://bugzilla.redhat.com/show_bug.cgi?id=1243655>
03:29 shubhendu joined #gluster
03:37 atinm joined #gluster
03:42 vmallika joined #gluster
03:51 arcolife joined #gluster
03:54 maZtah joined #gluster
03:57 nishanth joined #gluster
03:59 frankS2 joined #gluster
04:02 twisted` joined #gluster
04:04 itisravi joined #gluster
04:07 ppai joined #gluster
04:13 sakshi joined #gluster
04:15 vimal joined #gluster
04:16 kanagaraj joined #gluster
04:16 RameshN joined #gluster
04:18 billputer joined #gluster
04:18 Lee1092 joined #gluster
04:23 samsaffron___ joined #gluster
04:31 nbalacha joined #gluster
04:36 lezo joined #gluster
04:36 nishanth joined #gluster
04:40 surabhi joined #gluster
04:40 spandit joined #gluster
04:48 anil joined #gluster
04:50 craigcabrey joined #gluster
04:51 surabhi joined #gluster
04:55 ashiq joined #gluster
04:59 sadbox joined #gluster
05:02 hgowtham joined #gluster
05:05 ndarshan joined #gluster
05:06 ndarshan joined #gluster
05:08 karnan joined #gluster
05:08 gem joined #gluster
05:10 pppp joined #gluster
05:11 meghanam joined #gluster
05:14 deepakcs joined #gluster
05:15 soumya joined #gluster
05:24 smohan joined #gluster
05:29 jiffin joined #gluster
05:32 Manikandan joined #gluster
05:35 kdhananjay joined #gluster
05:38 Bhaskarakiran joined #gluster
05:42 hagarth joined #gluster
05:48 autoditac joined #gluster
05:55 redbeard joined #gluster
05:55 andras joined #gluster
05:58 Philambdo joined #gluster
05:58 andras I have a few remaining files left on brick after remove-brick operation. Does anyone know if it is safe to rsync data (without xatrs) from brick (to be removed) to gluster mount point?
05:59 andras I know it is not good to rsync to brick, but what about from brick?
06:03 aravindavk joined #gluster
06:04 andras it is strange why remove-brick did not clean-up and move all files. I got  setting xattrs on file failed -operation not supported. and no idea why.
06:06 atalur joined #gluster
06:15 saurabh_ joined #gluster
06:15 karnan joined #gluster
06:15 kshlm joined #gluster
06:18 shubhendu joined #gluster
06:20 shubhendu joined #gluster
06:22 jtux joined #gluster
06:25 PatNarcisoZzZ joined #gluster
06:27 glusterbot News from resolvedglusterbugs: [Bug 1228981] glusterfind throws syntax error while getting packaged on EL5 <https://bugzilla.redhat.com/show_bug.cgi?id=1228981>
06:28 nsoffer joined #gluster
06:28 nangthang joined #gluster
06:30 dusmant joined #gluster
06:32 shubhendu_ joined #gluster
06:38 vmallika joined #gluster
06:50 raghu joined #gluster
06:52 smohan joined #gluster
07:01 kshlm joined #gluster
07:07 mbukatov joined #gluster
07:10 cogsu joined #gluster
07:12 rjoseph joined #gluster
07:16 [Enrico] joined #gluster
07:18 meghanam joined #gluster
07:26 autoditac joined #gluster
07:28 Slashman joined #gluster
07:41 iamb joined #gluster
07:47 ctria joined #gluster
07:47 fsimonce joined #gluster
07:57 maveric_amitc_ joined #gluster
08:04 bharata-rao joined #gluster
08:04 Philambdo joined #gluster
08:09 Telsin joined #gluster
08:15 PatNarcisoZzZ joined #gluster
08:16 gildub joined #gluster
08:19 topshare joined #gluster
08:27 topshare joined #gluster
08:41 kshlm joined #gluster
08:46 elico joined #gluster
08:46 dusmant joined #gluster
08:48 ajames-41678 joined #gluster
08:49 shubhendu joined #gluster
08:49 ndarshan joined #gluster
08:50 tessier How much performance improvement is there between fuse and libgfapi for serving busy VM images?
08:51 tessier fuse is working fine but when I configure my testvm to use libgfapi the VM booted but immediately got IO errors and went read only.
08:55 gem_ joined #gluster
08:56 Peppard joined #gluster
08:58 glusterbot News from newglusterbugs: [Bug 1243753] Gluster cli logs invalid argument error on every gluster command execution <https://bugzilla.redhat.com/show_bug.cgi?id=1243753>
09:06 soumya_ joined #gluster
09:12 Manikandan joined #gluster
09:16 gem joined #gluster
09:22 dusmant joined #gluster
09:29 shubhendu joined #gluster
09:31 harish joined #gluster
09:33 yuusou joined #gluster
09:34 ndarshan joined #gluster
09:34 yuusou hey, I was wondering if someone could help me with some information I need to migrate some data.
09:34 Pupeno joined #gluster
09:34 Pupeno joined #gluster
09:35 yuusou I've got a replicated-distributed hardware based gluster 3.5 which will be migrated to a virtual 3.7 replicated-only.
09:36 yuusou this is a production environment so I need to rsync the files between them and gradually change mount points on single systems.
09:36 yuusou what client version do I use on these systems?
09:37 yuusou seeing that I need 3.5 and 3.7 mounted simultaneously.
09:37 ira joined #gluster
09:46 teknologeek joined #gluster
09:47 teknologeek hi all
09:48 teknologeek [2015-07-16 09:46:45.460087] E [dht-helper.c:1321:dht_inode_ctx_set] (-->/usr/lib64/glusterfs/3.5.4/xlator/cluster/distribute.so(dht_readdirp_cbk+0x2d3) [0x7f97d4681283] (-->/usr/lib64/glusterfs/3.5.4/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7f97d465b81e] (-->/usr/lib64/glusterfs/3.5.4/xlator/cluster/distribute.so(dht_inode_ctx_layou
09:48 teknologeek t_set+0x57) [0x7f97d465d7f7]))) 0-XXXX-dht: invalid argument: inode
09:48 glusterbot teknologeek: ('s karma is now -92
09:48 glusterbot teknologeek: ('s karma is now -93
09:48 glusterbot teknologeek: ('s karma is now -94
09:48 teknologeek any clue on this message ?
09:49 teknologeek my /var/log is full now :P
09:53 pppp joined #gluster
09:53 jmarley joined #gluster
09:54 sakshi joined #gluster
09:55 yuusou joined #gluster
09:56 hagarth nbalacha: ^^ any clue on this log message?
09:57 teknologeek :,(
09:57 teknologeek :'(
09:58 nbalacha Is this a dist-rep volume?
09:58 glusterbot News from newglusterbugs: [Bug 1243768] GlusterD crashes when management encryption is enabled <https://bugzilla.redhat.com/show_bug.cgi?id=1243768>
10:00 Philambdo1 joined #gluster
10:02 sakshi joined #gluster
10:10 sabansal_ joined #gluster
10:37 kkeithley1 joined #gluster
10:38 nbalacha teknologeek, this looks like something we fixed
10:38 kdhananjay joined #gluster
10:38 nbalacha teknologeek, let me tr to find the commit
10:43 LebedevRI joined #gluster
10:45 arcolife joined #gluster
10:59 soumya_ joined #gluster
10:59 glusterbot News from resolvedglusterbugs: [Bug 1243768] GlusterD crashes when management encryption is enabled <https://bugzilla.redhat.com/show_bug.cgi?id=1243768>
11:00 ira joined #gluster
11:14 _shaps_ joined #gluster
11:17 atinm joined #gluster
11:17 aravindavk joined #gluster
11:18 Romanr halo lads
11:19 rjoseph joined #gluster
11:21 tessier 2806 root      20   0  919488  64960   3904 S 153.8  0.1  73:08.78 glusterfs
11:21 tessier 153% cpu time makes me worried.
11:22 soumya_ joined #gluster
11:22 md2k why? you have 20 threads and usage of 1cpu +50%, maybe some self-heal operation in background, not something to worry about imho
11:22 tessier 20 threads?
11:22 tessier I'm scp'ing data into gluster from another system.
11:23 md2k about threads nvm, about load during scp'ing files to gluster, it fine, check logs for any errors or heal tasks
11:26 md2k you can use atop tool with flags '-f 1'  to see more information about processes, also you can check 'iotop' to see disk utilization by gluster process
11:27 md2k but in general 150% fpr multi-threaded applications something normal , and if you do upload bunch of file to gluster over its mountpoint it is fine to see more then 1 core is busy with something
11:28 dusmant joined #gluster
11:28 tessier Ok, thanks. I'm just really worried about performance. This thing is hosting VM images. I think it is already bandwidth constrained with a single gigabit connection. We are going to upgrade it to dual gigabit this weekend.
11:28 tessier Just running ls on the gluster mount takes a few seconds.
11:29 md2k many files per folder ?
11:29 tessier No, just a few big files.
11:32 yuusou I need to mount two gluster volumes from different versions (3.5 and 3.7) on a system, what client version do I use?
11:41 tessier Wow, this VM is booting incredibly slowly.
11:41 atinm joined #gluster
11:42 md2k did you check network/disk/cpu utilization by gluster during VM boot ?
11:42 md2k also you gluster volume options is default ?
11:44 gem joined #gluster
11:47 tessier cpu and disk utilization is minimal. Not sure how to check network...default options except for the recommended virt-store use case tuning recommended here: https://www.gluster.org/community/documentation/index.php/Virt-store-usecase#Applying_the_Tunables_on_the_volume
11:48 * tessier fires up iftop
11:50 tessier It says I'm only receiving at 234Mb/s.
11:50 yuusou are you using fuse or libgfapi? what hypervisors are you using?
11:50 tessier fuse. kvm.
11:51 tessier Tried libgfapi and the VM booted but immediately got io errors
11:51 yuusou fuse will probably be a huge bottleneck
11:51 tessier Yes, that's what I'm afraid of.
11:51 yuusou I'd recommend retrying libgfapi
11:52 yuusou or at least NFS
11:52 topshare joined #gluster
11:52 md2k for libgfapi there is advice in docs to set allow-insecure to n
11:52 md2k *on
11:52 tessier Yes, I did that.
11:52 yuusou do you have the allow unsafe flag?
11:52 tessier Yes, I do
11:53 yuusou and did you stop and start the volume?
11:53 tessier Yes, I did
11:53 md2k thrn we need gluster guru maybe, not have enough knowledge with VMs over gluster
11:54 md2k also you edit volume file and set rpc-auth-allow-insecure to on also ?
11:54 yuusou I've got ovirt with gluster37 using libgfapi and working perfectly, no issues with VMs
11:54 tessier md2k: Ye
11:54 tessier s
11:55 tessier yuusou: That's good to hear. Decent performance? Limited by hard drives or network and not gluster?
11:55 md2k tessier: also you set all options from section `Tunables` ?
11:55 yuusou performance is good, no limitations, it's perfectly usable.
11:56 natarej speaking of performance, i'm about to set it up in the lab
11:56 yuusou 2x 10gbit connections, 10k disks raid 5
11:56 tessier md2k: Yes
11:56 natarej what sort of throughput do you guys think i'm going to get on 1GbE?
11:56 yuusou big files with jumbo frames you should be able to fill up the link
11:56 tessier natarej: I'm using 1Gb. I expect close to 100MB/s networkwise if the disks can push it.
11:56 natarej nice
11:57 natarej i thought there would be a lot more overhead
11:57 * tessier is using jumbo frames too
11:57 Romanr hey, what would be best volume performance settings for VM storages, where min crated file is 15G
11:57 md2k tessier: dunring VM boot check with atop -f 1 , disk utilization, it not reach 100% IO ?
11:57 tessier RX packets:91557534 errors:0 dropped:15567445 overruns:0 frame:0
11:58 tessier Ah...I think I've found something.
11:58 Romanr cluster.stripe-block-size default is 128K, may be there is a point to make it a bit bigger? 512K ie
11:58 glusterbot News from newglusterbugs: [Bug 1243812] [Backup]: Crash observed when keyboard interrupt is encountered in the middle of any glusterfind command <https://bugzilla.redhat.com/show_bug.cgi?id=1243812>
12:00 yuusou dropped packets, are you sure you've got jumbo frames everywhere?
12:00 tessier yuusou: I just checked and yes, jumbo frames enabled on both gluster nodes and the KVM machine.
12:02 yuusou both on the same vlan?
12:02 tessier Yes
12:02 atinm joined #gluster
12:02 Romanr and jumbo frames on switches is on?
12:02 topshare joined #gluster
12:02 tessier Romanr: Yes.
12:03 yuusou no mtu defined on the switch or on the vlan interface?
12:03 Romanr ifconfig to the studio pls
12:04 yuusou well, try a ping with large packages and see if you get a reply
12:04 tessier Romanr: ?
12:04 Romanr copy paste ifconfig output here
12:05 dusmant joined #gluster
12:07 tessier Too much for the channel. Here it is: http://pastebin.com/BGN2z2vv
12:07 glusterbot Please use http://fpaste.org or http://paste.ubuntu.com/ . pb has too many ads. Say @paste in channel for info about paste utils.
12:08 tessier Romanr: I've got ifconfig from both of my gluster nodes, my kvm machine, and the output of "show vlan" on my switch (procurve).
12:09 Romanr hm, its fine with mtu
12:10 jtux joined #gluster
12:10 Romanr do you have option rpc-auth-allow-insecure on in glusterd.vol file?
12:11 unclemarc joined #gluster
12:11 tessier This VM is still booting. 30 minutes to boot a VM is nuts.
12:11 tessier Romanr: Yes, I do.
12:11 tessier Romanr: Although that would affect more libgfapi than general performance wouldn't it?
12:12 Romanr what kind of VM are you trying to install?
12:12 Romanr OS?
12:12 tessier I am migrating VMs from Xen to KVM. A combination of CentOS 6 and 6.
12:13 Romanr and hypervisor is pure kvm or you use something like proxmox/
12:13 tessier Pure KVM. The hypervisor machine is newly installed CentOS 7
12:14 Romanr and no connections errors?
12:14 tessier No.
12:15 Romanr and if you boot up with libgfapi you've got io errors?
12:15 tessier Yes.
12:16 _shaps_ left #gluster
12:16 Romanr try to ping -s 5000 from glusterfs server to kvm hypervisor
12:16 yuusou did you disable selinux and firewalld when trying libgfapi?
12:16 tessier yuusou: Yes, selinux is in permissive mode.
12:16 tessier Romanr: Trying...
12:16 * Romanr thinks that every one disables that sexlinux on centos
12:16 Romanr :D
12:17 tessier Romanr: I don't. :) It is disabled for now on these machines but I have a ton of other machines running SELinux. It has saved our butts a few times now.
12:17 * yuusou has selinux running on a mail and dns server
12:17 tessier 39 packets transmitted, 39 received, 0% packet loss, time 38151ms
12:17 tessier rtt min/avg/max/mdev = 0.300/0.344/0.389/0.029 ms
12:17 tessier That's with -s 5000
12:17 Romanr now try the same with -f switch
12:18 tessier 300886 packets transmitted, 300885 received, 0% packet loss, time 17808ms
12:18 tessier rtt min/avg/max/mdev = 0.040/0.045/3.477/0.008 ms, ipg/ewma 0.059/0.045 ms
12:18 Romanr than your problem is not netwrorking related.
12:19 nsoffer joined #gluster
12:19 Romanr have you changed any options for volumes?
12:20 yuusou you can't rule that out, he had package drops. have you cleared the counters see if there are still drops?
12:20 tessier Nothing other than the virt-store options as I described earlier.
12:21 anrao joined #gluster
12:22 tessier [root@disk07 brick]# /sbin/ifconfig eth1|grep "RX packets"; sleep 10; /sbin/ifconfig eth1|grep "RX packets";  RX packets:98864403 errors:0 dropped:16206556 overruns:0 frame:0 RX packets:98865778 errors:0 dropped:16206603 overruns:0 frame:0
12:22 nbalacha joined #gluster
12:22 tessier hrm...that formatting got hosed. In 10 seconds it dropped 47 packets.
12:22 tessier both of my gluster disk nodes seem to be doing about that same rate
12:23 yuusou that's a lot of drops
12:23 yuusou reset mtu back to 1500 for now
12:23 Romanr yuusou, dropped packets are normal. those could be anything. ie ipv6 frames if server is not v6 configured etc
12:24 Romanr if you don't use that link (no traffic there is generated by ur servers) still counter rises?
12:25 Romanr if not, than should check rj45
12:25 tessier Romanr: Not sure...that would be hard to test right now. I would have to shutdown my VMs and everything.
12:25 tessier Yes, I am considering swapping out network cables just in case.
12:26 Romanr are there any errors on switch interfaces?
12:26 yuusou dropped packets aren't normal unless there's something polling his server for something it doesn't have
12:28 yuusou which is highly suspicious anyway
12:28 tessier Indeed
12:28 * tessier checks switch interfaces for errors
12:29 glusterbot News from newglusterbugs: [Bug 1243815] DHT: REBALANCE - Rebalance crawl on a directory will never visit peer directories if fix-layout fails for any of the descendant directories <https://bugzilla.redhat.com/show_bug.cgi?id=1243815>
12:30 yuusou anyway, back to the really difficult question that's on everyone's mind: if you need to mount two gluster volumes from different versions (3.5 and 3.7) on a system, what client version do you use?
12:31 itisravi_ joined #gluster
12:31 tessier errors on the switch except for: Giants Rx       : 9,606,906
12:31 aravindavk joined #gluster
12:33 tessier However, that seems to be expected behavior.
12:33 tessier Jumbo packets always increment the giant counter, the manual says so.
12:34 yuusou vlan <id> jumbo
12:34 tessier jumbo is already set
12:34 tessier 802.1Q VLAN ID Name         Status       Voice Jumbo
12:34 tessier -------------- ------------ ------------ ----- -----
12:34 glusterbot tessier: ------------'s karma is now -2
12:34 glusterbot tessier: ----------'s karma is now -2
12:34 glusterbot tessier: ----------'s karma is now -3
12:34 glusterbot tessier: ---'s karma is now -1
12:34 glusterbot tessier: ---'s karma is now -2
12:34 tessier 5              cpc-san  Port-based   No    Yes
12:34 Romanr there should not be any errors on interfaces. swap cables.
12:35 davidbitton joined #gluster
12:35 tessier I will. But it seems quite unlikely that both of my disk servers have a bad cable.
12:35 Romanr if they comes form same supplier, i would not be surprised
12:37 Romanr JoeJulian, ndevos _maserati_ seems like i have to close my bug report. d8 installs fine with both scsi and sata contras. only virtio has problems, so it seems like it is proxmox qemu related, as d8 installed well on ovirt systems
12:38 tessier Well, I've been working on this all night and the sun is coming up. I need some sleep. Thanks a lot for the help! I'll try libgfapi again after I have had some rest.
12:38 yuusou good luck tessier
12:39 yuusou are gluster clients backwards compatible?
12:42 Romanr there is a problem between 3.5.x clients and >= 3.6.x servers, as there was self healing totally rewritten
12:42 Romanr so if you not going to update clients, you have to set some params for the volume
12:44 topshare joined #gluster
12:45 topshare joined #gluster
12:46 yuusou I was thinking of updating the clients and use 3.7 client with 3.5 server end
12:46 topshare joined #gluster
12:47 yuusou and the new servers will be 3.7 of course
12:47 topshare joined #gluster
12:53 pdrakeweb joined #gluster
12:54 Romanr yuusou, be sure to check this one http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6
12:54 Romanr same with 3.7
12:55 plarsen joined #gluster
12:57 smohan_ joined #gluster
12:57 B21956 joined #gluster
12:58 wkf joined #gluster
13:00 smohan- joined #gluster
13:01 glusterbot News from resolvedglusterbugs: [Bug 1242913] Debian Jessie as KVM guest on GlusterFS backend <https://bugzilla.redhat.com/show_bug.cgi?id=1242913>
13:05 lpabon joined #gluster
13:07 gem joined #gluster
13:13 shyam joined #gluster
13:17 mpietersen joined #gluster
13:17 calisto joined #gluster
13:19 ekuric joined #gluster
13:20 jotun joined #gluster
13:23 jmarley joined #gluster
13:25 georgeh-LT2 joined #gluster
13:28 Romanr okay. going out of office. was nice to socialize
13:28 * Romanr is away: Really :-)
13:33 topshare joined #gluster
13:34 ashiq joined #gluster
13:34 hamiller joined #gluster
13:38 Bhaskarakiran joined #gluster
13:39 dijuremo joined #gluster
13:40 vmallika joined #gluster
13:42 cholcombe joined #gluster
13:44 gem joined #gluster
13:48 TheCthulhu1 joined #gluster
13:48 shyam joined #gluster
13:59 Saravana_ joined #gluster
13:59 kanagaraj joined #gluster
13:59 arcolife joined #gluster
13:59 dijuremo joined #gluster
13:59 marbu joined #gluster
14:00 dgandhi joined #gluster
14:00 topshare joined #gluster
14:00 Saravana_ joined #gluster
14:01 Saravana_ joined #gluster
14:03 Saravana_ joined #gluster
14:06 anrao joined #gluster
14:15 shyam joined #gluster
14:21 mbukatov joined #gluster
14:23 anrao joined #gluster
14:25 vmallika joined #gluster
14:27 Gill joined #gluster
14:28 gem joined #gluster
14:32 Vortac joined #gluster
14:39 soumya_ joined #gluster
14:45 Trefex joined #gluster
14:47 nbalacha joined #gluster
14:48 mpietersen joined #gluster
14:54 dgbaley joined #gluster
14:55 Gill joined #gluster
14:59 Gill_ joined #gluster
15:04 hagarth joined #gluster
15:04 Gill joined #gluster
15:10 Gill joined #gluster
15:11 jmarley joined #gluster
15:20 vmallika joined #gluster
15:25 sankarshan_ joined #gluster
15:26 shyam joined #gluster
15:29 Gill joined #gluster
15:37 jobewan joined #gluster
15:37 tessier Does anyone do any gluster consulting? I'm wondering if anyone would like to help me investigate my speed problems. We're up against a daedline and it might take me too long to sort this out myself.
15:38 aravindavk joined #gluster
15:43 calavera joined #gluster
15:44 nsoffer joined #gluster
15:45 tessier I am scp'ing in files from another server and scp says: ./corpsites2.img: Read-only file system
15:45 tessier gluster wouldn't let it write
15:46 tessier But scp successfully went onto the next file
15:47 kanagaraj joined #gluster
15:49 hagarth tessier: http://gluster.org/consultants/ might be a good place to start
15:51 tessier Yeah, I'm looking at that now...
15:58 tessier They're all in Europe...hrm...
15:58 _maserati_ Romanr, glad that worked for you!
16:04 tessier They're all asleep/out of the office by now.
16:06 tessier ah....I've found some errors in the logs...
16:06 tessier [2015-07-16 16:03:54.516378] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/a8ab2993bb1e793a1a436d8ed92d9c8b.socket failed (Invalid argument)
16:06 PatNarcisoZzZ joined #gluster
16:08 tessier gluster native mount == fuse, correct?
16:09 dijuremo joined #gluster
16:20 shyam joined #gluster
16:22 tessier Heard from one...they are booked solid two weeks out...another's auto-reply says will get back to me within two business days...
16:27 tessier I must be doing this wrong. Because I've setup gluster twice on different hardware and get poor results each time.
16:28 calavera_ joined #gluster
16:28 cyberswat joined #gluster
16:41 shyam joined #gluster
16:45 jwd joined #gluster
16:45 TvL2386 joined #gluster
16:50 jwd joined #gluster
16:59 calisto joined #gluster
17:01 tessier I'm only getting 33MB/s writing to my gluster volume. I really expect more like 100MB/s. Maybe 50 if it has to write a separate copy to each of my two replicas.
17:01 tessier If I add any other load to the filesystem it slows way down and eventually produces errors.
17:04 tessier I've got jumbo packets enabled everywhere, full duplex 1000MB/s on all links...
17:05 Philambdo joined #gluster
17:05 Gill joined #gluster
17:13 mpietersen joined #gluster
17:14 JoeJulian Are your writes filling up packets?
17:14 JoeJulian wait.. "any other load" slows stuff down? That's not right.
17:15 JoeJulian Nothing obvious in top or iotop?
17:16 JoeJulian And yes, the client writes to *all* replica, so you should see bandwidth divided by replica count.
17:19 7YUAAAEHY joined #gluster
17:21 Romanr tessier, wazup. what ciphers are used for scp? try without any
17:21 Romanr _maserati_, i'm glad also! :D
17:23 swebb joined #gluster
17:26 bene2 joined #gluster
17:27 craigcabrey joined #gluster
17:29 jdossey joined #gluster
17:29 turkleton joined #gluster
17:32 turkleton Hey folks. How's it going?
17:35 Gill joined #gluster
17:46 Romanr JoeJulian, _maserati_ but really.. there is still a question: why virtio drivers work on local disks
17:46 Romanr :)(
17:47 Romanr libvirt issue?
17:47 Romanr i mean libgfapi
17:47 tessier JoeJulian: Filling up packets? How would Iknow? Nothing obvious in top or iotop.
17:48 tessier Disk utilization is low, glusterd often uses 150% cpu during a copy but I am told that is normal.
17:48 tessier Romanr: It isn't cpu utilization. Even if I just copy within gluster it is really slow.
17:48 tessier I was doing a copy and scp said "read only file" and quit. I'm pretty sure that is related to gluster.
17:48 tessier Two of my VMs currently running on gluster hung.
17:49 tessier I've long suspected something was borked here but could never nail it down.
17:49 Romanr what gluster version u use?
17:50 tessier 3.7.0 everywhere
17:50 Romanr why don't try latest one
17:50 tessier I thought this was the latest stable...
17:50 Romanr 3.7.2 afaik
17:50 tessier Yeah, I have that in my repo...
17:50 Gill_ joined #gluster
17:51 tessier Upgrading...
17:51 Romanr and check change logs also
17:51 Romanr may be your bug was fixe
17:51 tessier I hate upgrading without knowing what the real problem is but I'm out of other ideas.
17:54 bfoster joined #gluster
17:55 Gill joined #gluster
17:55 PatNarciso tessier, what is your underlying filesystem?
17:55 tessier PatNarciso: xfs
17:56 * PatNarciso pulls up a seat at tessier's table.
17:56 tessier :)
17:56 tessier ugh...I think the VMs just hung again...
17:57 Romanr and there is no way for you to try with ext4? i had a lot of trobles with xfs
17:57 Romanr while xfs really recommended by gluster
17:57 tessier I was told xfs was THE thing to use. All of the docs seem to say that too.
17:57 Romanr http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6
17:57 Romanr oh wrong link
17:58 turkleton What's the best way to replace a replicated node in the event of a total catastrophic event where a node is lost? Is there a recommended way to replace the dead brick? I saw that replace-brick was deprecated in 3.6, should I have done an add/remove instead?
17:58 Romanr and what are your xfs mount options/
17:58 Romanr tessier,
17:59 PatNarciso from what I understand, gluster is attr heavy.  and... at 'some point' the the performance seems to be tanking... BUT -- accessing the underlying xfs mount is swift.
18:00 Romanr you never know what causes errors when u use so many layers
18:01 * PatNarciso nods
18:03 PatNarciso fwiw: I digg this pdf.  http://xfs.org/images/d/d1/Xfs-scalability-lca2012.pdf
18:04 Gill joined #gluster
18:07 autoditac joined #gluster
18:08 Romanr guys, does anyone know, if virt-manager (the gui from kvm vm createion) supports libgfapi out of the box/
18:09 Romanr if yes, i will smash all the proxmox nodes and go on with pure debian + kvm
18:10 jermudgeon Romanr: YMMV
18:10 jermudgeon I really like it, but I’ve found its libgfapi support to be… problematic.
18:11 jermudgeon It might be fine if you compile everything from the latest sources, but as it currently stands…
18:11 jermudgeon what are going to run virt-manager on?
18:11 jermudgeon I mean, what ver of debian?
18:11 jwd joined #gluster
18:11 jermudgeon I’m running on CentOS 7, but I was actually thinking of trying proxmox… convince me otherwise?
18:11 bfoster joined #gluster
18:12 Romanr debian 8
18:13 Romanr hah. i just with help of glusterfs devs and community found out, that latest proxmox's (not beta) qemu is not compatible with debian 8 virtio drivers. i was trying to debug this problem for 2 months. debian8 installation just failed at random step. if finished, i had unusable system with ro fS sometims.
18:13 Romanr all the binaries were corrupted
18:14 Romanr if i choose another driver (not virtio) thigs go fine
18:14 Romanr and this is only with d8 and d8 testing
18:14 Romanr c7, ubuntu 14.04, d7 - ran fine
18:15 Romanr and had absolutely no help from proxmox community nor devs. they just said - well, it installs ok on local disk, the problem is on glusterfs side. and i was pretty big PITA for gluster community for 2 months in mailing list and few days here
18:16 Romanr don't know how could they endured me :D
18:17 tessier PatNarciso: Are you saying you are having problems similar to mine?
18:18 PatNarciso I don't use my gluster for vm's, but I do have odd performance issues.
18:19 Romanr tessier, what are your xfs mount options?
18:19 Romanr oh, wait
18:19 Romanr u answered that one yeterday/
18:19 Romanr yesterda
18:19 Romanr yesterday FFS
18:19 PatNarciso ffs; nice. haven't used that one yet.
18:20 natarej_ joined #gluster
18:20 tessier Romanr: No special xfs mount options.
18:21 tessier block I/O error in device 'drive-ide0-0-0': Read-only file system (30)
18:21 tessier Now my kvm is spewing that all over the console.
18:22 Romanr tessier, then.. you have to change this
18:22 Romanr i'll give you a link
18:22 Romanr wait
18:23 Romanr i can say, that i know ur problem
18:23 Romanr proxmox has the same, but only when u create a VM
18:23 tessier 10.0.1.12:/disk07b on /gluster/disk07b type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
18:23 Romanr there goes kind of lock time out
18:23 tessier Those are the mount options.
18:23 tessier But I just mount it with:  mount -t glusterfs 10.0.1.12:/disk07a /gluster/disk07a/
18:23 tessier So those are defaults
18:24 tessier Even when not using VMs, just doing normal stuff inside the mount, it is slow.
18:25 Romanr https://github.com/purpleidea/puppet-gluster/blob/master/manifests/brick.pp#L261 + definitely set noatime,inode64,nodiratime,allocsize=64k
18:26 Romanr you can even try without those things in link
18:26 Romanr just last 4 options
18:26 Romanr and see then if something changed
18:26 Romanr if nothing help, give ext4 a try. i do not use xfs at all with gluster. all on ext5
18:26 Romanr ext4*
18:27 rjoseph joined #gluster
18:27 PatNarciso ... does noatime imply nodiratime?
18:27 Romanr never tried to figure out. just know that xfs with default options sux
18:27 PatNarciso amen.
18:28 Rapture joined #gluster
18:28 Romanr and don't waste my time on this FS, as it implies a guy who really know what he will need and all those million options and performance tuning
18:28 JoeJulian Meh, ext4 is a mess. Stick with xfs.
18:29 Romanr for me ext4 worked best with gluster on 4 nodes with proxmox
18:29 tessier Moving all of my data would be a real pain. I'll stick with xfs until I know for sure it is a problem. If JoeJulian and gluster.org recommend xfs then it's probably pretty good.
18:29 jermudgeon xfs has definitely been recommended; I think I’m running a mix atm
18:29 JoeJulian If you look at the source for both, you'll pick xfs.
18:30 tessier JoeJulian: Do you know if any of those xfs mount options are likely to help?
18:30 PatNarciso 'probably pretty good'.  I need this on a t-shirt.
18:30 JoeJulian The volume going read-only is probably a loss of connection to a brick, resulting in a quorum failure. Check the client log.
18:31 Romanr JoeJulian, i can't stick with it for a simple reason: if i start a vm on proxmox with gluster backend i fail with some kind of lock time-out. just right out of the box. and how would i know that i must set allocsize=64k to get it working? its just not able to create a qcow file on the fly without this option in fstab
18:31 JoeJulian I've never set that.
18:32 calisto joined #gluster
18:33 JoeJulian Yeah, double checked. The default allocsize is 64k.
18:34 JoeJulian Thought some numbers snob lists it in the kernel docs as KiB. I'm never going to adopt that silly Kibbi crap.
18:34 Romanr JamesG, here its http://forum.proxmox.com/threads/17462-SOLVED-Linux-NFS-or-GlusterFS-Server-not-creating-sparse-files?highlight=nfs+glusterfs
18:34 Romanr oops, JoeJulian
18:35 PatNarciso JoeJulian- xfs related- any suggestions for updating an xfs partition that wasn't formated with -i size=512 -n size=8192?  (it had been formated with defaults).
18:35 PatNarciso and... you know the story... too much data, would take to long to relocate... etc.
18:35 shyam turkleton: Check http://review.gluster.com/#/c/8503/3/doc/admin-guide/en-US/markdown/admin_replace_brick.md for an answer to replacing bricks
18:37 Romanr tessier, you could just try those options, it won't hurt anyway.
18:39 Romanr but well this could be only proxmox related btw.
18:39 Romanr they use pretty old kernel
18:39 Pupeno joined #gluster
18:39 Romanr 2.6.32-39
18:39 JoeJulian Hmm, ok, re-reading the kernel doc I do see that it's a dynamic behavior unless allocsize is set. Odd that it has a default if it's not used. I assume that means you could just add "allocsize" instead of "allocsize=64k".
18:41 * Romanr thinks that GlusterFS for qemu VM-s should have separate line in docs or at least somehow mark, that default xfs mount with qemu VM-s = no go
18:42 Pupeno_ joined #gluster
18:43 Romanr by the way, i've got one replica volume with xfs 100G :)) didn't see any differences in performance... and afaik when xfs partition will be ca 90% filled it slows down like hell.. or was it fixed?
18:43 Dave joined #gluster
18:43 Romanr didn't see any difference after i set all the options i mean
18:43 hchiramm_home joined #gluster
18:44 ndevos Romanr: ah, maybe you were having issues with sparse files on xfs? there was an issue where xfs was too agressively predicting future usage and allocating blocks for those sparse areas
18:45 JoeJulian Sounds like that's what it is.
18:45 Romanr yep. and with proxmox i still have these issues
18:45 Romanr until i set allocsize=64k
18:46 ndevos it depends on what kernel (+patches) you use, but yes, gluster/strip had issues with that too, and setting allocsize disables the prediction xfs tries to do (I think it was something like that)
18:46 JoeJulian Right, but either way you're using the same kernel.
18:47 JoeJulian That's what https://www.kernel.org/doc/Documentation/filesystems/xfs.txt says
18:47 ndevos but, that has been fixed now, and current kernels should have a better prediction and should not allocate blocks for holes too much
18:48 turkleton shyam: Is rebalance recommended over heal even if it's only replicate and not distribute-replicate?
18:48 shyam If it is only replicate then rebalance is not needed
18:48 Romanr those are details... there are a lot of guys who would like to use proxmox (its community is pretty huge) and when fail with such setup (proxmox+gluster+xfs) they will just switch to something else :) just add a line to ur docs
18:49 Romanr i don't want to say, that xfs is bad. i just want to say, it won't work with some solutions
18:49 ndevos Romanr: update the kernel and it should work ?)
18:49 ndevos uh, :)
18:49 Romanr haha :D you can't update kernel on proxmox
18:49 JoeJulian I'm getting less and less impressed with proxmox every day.
18:49 Romanr haha JoeJulian  :D
18:49 Romanr proxmox uses OpenVZ kernel
18:49 Romanr and OpenVZ is dead.
18:50 JoeJulian It doesn't use the debian kernel?
18:50 Romanr no
18:50 shyam turkleton: see the section "##Replacing brick in Replicate/Distributed Replicate volumes" in the same md file that talks about only replicate replacement (bottom half of the document)
18:50 Romanr so next version (proxmox 4, at beta atm) will use KVM + LXC and use native debian 8 kernel
18:50 Romanr it will be fine then
18:50 turkleton Awesome, thanks. For some reason the page keeps auto-scrolling me back to the top. Navigation seems a bit wonky.
18:50 JoeJulian Oh, I thought you were on 4.
18:50 natarej_ with the disperse translator, are entire files still stored on bricks?
18:50 Romanr nope. I'm on 3.4
18:51 ndevos natarej_: nope, disperse splits the files in (encoded) chunks and stores those chunks
18:51 natarej_ ndevos, thanks
18:52 JoeJulian natarej_: http://www.gluster.org/community/documentation/index.php/Features/disperse#Detailed_Description
18:53 turkleton It looks the replacing brick there is done by creating a new directory and removing it to start an autoheal. Is there an issue with running a full heal instead since the new brick is blank?
18:54 Romanr tessier, don't you have some free space for test volumes in ur lvm ?
18:55 JoeJulian turkleton: If you have a large distribute portion to your volume, a full heal would require walking the entire directory tree which might exceed the needs of that one brick significantly. This could cause the self-heal to actually take longer than needed. If you just have a 2 brick replica 2 then it doesn't matter.
18:56 turkleton In this case, it is just a 2 brick replica 2. It sounds like a full heal is fine in that case (and essentially what would be initiated by the metadata change from the directory creation/deleteion), right?
18:58 Romanr jobewan, ndevos sorry to interrupt you... but could you please remind me what was the command to add a volume to predefined group ?
18:58 Romanr JoeJulian, *
18:59 ndevos Romanr: what do you mean with "predefined group"?
19:00 Gill_ joined #gluster
19:00 Romanr predefined group of settings
19:01 Romanr i just have some in /var/lib/glusterd/groups
19:01 Romanr but i don't remember the gluster volume set command. ..
19:01 ndevos ah, right, try: gluster volume help
19:01 * ndevos would do that too, but isnt logged in on a system with the gluster command atm
19:02 Romanr tried...
19:02 Romanr even tried gluster volume set help
19:03 Romanr lol
19:03 Romanr it was gluster volume set volname group groupname
19:03 Romanr :D:D:D
19:03 Romanr too much wine
19:05 ndevos hmm, that option is not documented? or I'm blind
19:05 Romanr may be not ;-)
19:06 ctria joined #gluster
19:07 * Romanr knows about glusterfs and its compatibility with 3rd parties a lot already... and only with 2 months of debugging an issue that was not gluster directly related...lol
19:07 Romanr now it comes out that I even use undocumented things
19:08 * Romanr thinks he's cool
19:08 JoeJulian ndevos: I think it's not. It was added for RHS, I think, and is in the RHS docs.
19:08 Gill joined #gluster
19:08 Romanr thats why i've got +x here :D:D
19:09 ndevos JoeJulian: yes, I know its in the RHS docs, but I would assume the option is documented in the help text
19:09 Romanr or +v .. don't remember which one
19:10 Romanr ok. now I've got fully adopted for glsuterfs and VMs an XFS partition.. will try it out someday :)
19:11 JoeJulian ndevos: I got the impression it was added as a last-minute, "We need this for XYZ customer asap." and it was just thrown in.
19:12 JoeJulian It appeared in the code like a few days before a RHS release.
19:12 ndevos JoeJulian: I think the oVirt devs needed it, they did not want to set many options, only one commands to set them
19:13 JoeJulian Seems likely.
19:17 ndevos if I read the change correctly, the cli reads the file and passes the options on to glusterd, thats a funky way of doing things - https://github.com/gluster/glusterfs/commit/35d178
19:17 Pupeno joined #gluster
19:25 Romanr but.. it works and i like it :D
19:25 Romanr by the way.. what language is used to write the glusterfs?
19:26 ndevos C, of course, not C++
19:26 glusterbot ndevos: C's karma is now 6
19:26 Romanr ah damn.. lol. it was discussed also few days ago
19:26 ndevos and some Python and shell bits too :)
19:27 Romanr i'm just under heavy impression that we've managed to find the cause of my problem...
19:29 hchiramm_home joined #gluster
19:30 glusterbot News from newglusterbugs: [Bug 1243991] "gluster volume set <VOLNAME> group <GROUP>" is not in the help text <https://bugzilla.redhat.com/show_bug.cgi?id=1243991>
19:31 Romanr deeen I wanted to add this one :D:D:D
19:31 * ndevos high fives _o/o_ glusterbot
19:34 natarej_ Romanr, how you going with D8?
19:35 Romanr nadley, bug closed. not glusterfs related. problem is between poxmox's qemu/libgfapi version and new D8 virtio drivers. as soon as i change disk driver to something but virtio, all work fine.
19:35 Romanr natarej_, *
19:36 Romanr whats wrong with those auto completions ...
19:36 redbeard joined #gluster
19:36 natarej_ alphabetical
19:36 Romanr i did press TAB twice! :D
19:39 Romanr is libgfapi is pure GF's thing?
19:39 Romanr in this case its up to proxmox qemu version only
19:42 tessier JoeJulian: There's lots of stuff in the client log but I don't know what to make of it. http://fpaste.org/245139/07569514/ Lots of errors. It looks like it should be a network problem not a gluster problem but if its a network problem I can't figure out where...maybe I need to swap out this switch....
19:46 JoeJulian 2-disk07b-client-0: remote operation failed: Transport endpoint is not connected
19:46 JoeJulian Kind-of a red flag there that the TCP connection cannot be made to the first brick.
19:47 Romanr firewall?!
19:47 calavera joined #gluster
19:49 JoeJulian Firewall or server not up or not responding seem the most likely possibilities.
19:50 Romanr tessier, just do iptables -F on servers
19:50 Romanr and test
19:51 turkleton JoeJulian: In the case we were talking about earlier, it is just a 2 brick replica 2. It sounds like a full heal is fine in that case (and essentially what would be initiated by the metadata change from the directory creation/deleteion), right?
19:53 tessier The server is up. There are no iptables. If there were I would think it wouldn't work at all.
19:53 tessier In fact the iptables module isn't even loaded on either machine.
19:54 tessier I have copied 600G of data and booted 6 VMs on this setup.
19:54 tessier I'm going to swap out the switch tonight. Just to be sure. Because nothing else adds up.
19:57 Romanr what switch are u using
19:58 tessier Procurve J4904A. It's a somewhat old switch but handles gigabit perfectly fine.
19:58 tessier Or should. All of my others do. I'm wondering if this one is a problem.
19:58 Romanr JoeJulian, how does peering works on gluster? aint its multicast?
20:01 Pupeno joined #gluster
20:01 tessier I can do ping -s 5000 -f 10.0.1.12 and get: 20748 packets transmitted, 20748 received, 0% packet loss, time 7331ms
20:01 tessier Shouldn't that really prove it isn't the layer2?
20:03 tessier I have been ssh'd into all of the involved servers also and never had any connectivity problems.
20:10 mpietersen joined #gluster
20:11 tessier Hmm...I don't think it could be the switch because my other gluster setup on the other switch has the same problem.
20:11 tessier It's got to be something I've setup wrong here.
20:11 bennyturns joined #gluster
20:17 gem joined #gluster
20:19 tessier [2015-07-16 20:17:19.418123] W [fuse-bridge.c:2273:fuse_writev_cbk] 0-glusterfs-fuse: 15375531: WRITE => -1 (Read-only file system)
20:19 tessier I am still getting these spewing all over the logs. Yet I can write to both of my bricks.
20:21 tessier I can touch a file on the client mounted volume and it appears in the backend. So why is it saying read-only?
20:21 Gill joined #gluster
20:27 Gill_ joined #gluster
20:28 Romanr tessier, copypaste here gluster peer status
20:29 Romanr and gluster pool list
20:30 jwd hi there
20:30 Romanr ehlo
20:30 jwd is there anything to monitor the health of gluster?
20:30 JoeJulian turkleton: Correct.
20:31 Romanr jwd, gluster volume status :)
20:31 JoeJulian Romanr: Nothing's multicast in gluster. I argued for that a long time ago, but nothing came of it.
20:32 Romanr JoeJulian, tnx
20:32 Romanr JoeJulian, not related question: will bot automatically give me +v :))?
20:32 JoeJulian tessier: "Yet I can write to both of my bricks" You're not supposed to write to bricks. That would be like using dd to write to the block device of a filesystem.
20:33 Pupeno joined #gluster
20:33 Romanr sorry, i'll afk for a while.
20:33 * Romanr is away: Really :-)
20:33 tessier JoeJulian: I mis-spoke: I'm not writing directly to bricks.
20:33 tessier I'm writing to the mounted filesystem on the client as intended
20:33 jmarley joined #gluster
20:33 tessier But when I do touch foo I can then peek into the backend and see foo appear on each replicant
20:34 jwd nvm a simple google question got me what i needed
20:34 JoeJulian Mmkay. Sounds like the connection may be intermittent.
20:34 tessier Romanr: Stand by for the pastes...
20:34 JoeJulian jwd: Romanr  answered you, too.
20:34 jwd yep that i seen
20:35 JoeJulian Romanr++
20:35 glusterbot JoeJulian: Romanr's karma is now 1
20:35 tessier Romanr: http://fpaste.org/245156/43707892/
20:43 tessier Romanr: Everything look normal in that output?
20:44 Romanr teknologeek, to me it seems like ok. every other server shows the same?
20:44 Romanr tessier, *
20:44 tessier Yep
20:44 tessier I dont see any layer 2 link bounces anywhere...
20:45 Romanr here is another advice.. just to see that there are really no problems with networking
20:46 Romanr install iperf and run it between client and every server
20:46 Romanr and try it multiple times
20:46 tessier Did that a few weeks ago when I first suspected these issues...running it again now...
20:46 JoeJulian I'll bet that it works.
20:46 Romanr hm
20:46 JoeJulian Because I suspect that it goes read only only when it fails.
20:47 JoeJulian Then it recovers and you're left wondering why there are errors in the logs.
20:47 tessier It never seems to fully recover as it is always slow.
20:47 JoeJulian It's probably healing
20:47 JoeJulian gluster volume heal $vol info
20:48 JoeJulian ... and if whatever is causing the disconnect keeps happening, those heals are never going to finish.
20:49 JoeJulian Also, peer status doesn't really tell much.
20:49 tessier Number of entries: 0
20:49 tessier That means it isn't healing right?
20:49 Romanr considering dropped packets on your server's NIC i'd start with L1 and then went higher... means: cables, switches etc..
20:49 JoeJulian That's just the management daemon's connections, nothing to do with the actual volume.
20:50 JoeJulian Check 'gluster volume status'
20:50 Romanr i just wanted to be sure that there is connection between servers at all :)
20:50 JoeJulian +1
20:50 tessier I have two volumes...diak07a doesn't seem to be healing...But I just ran gluster volume heal disk07b info and it seems to be hung...
20:52 Romanr if you've got some electric-magnetic devices next to you servers or in the way of copper cables, these could cause problems (we've had to switch to fiber on some places, as I work on company which translates TV and Radio station on air)
20:52 tessier Ah, it finally returned. Number of entries: 0
20:53 tessier [2015-07-16 20:50:22.576740] W [fuse-bridge.c:2273:fuse_writev_cbk] 0-glusterfs-fuse: 16113551: WRITE => -1 (Read-only file system)
20:53 Romanr another way - just break everything and start from zero
20:53 tessier Still have tons of that streaming in the logs. What does it mean? I can write to my volume.
20:56 tessier I bet if I do a tcpdump I'm going to see a disaster on the wire.
20:58 Gill joined #gluster
21:02 calavera joined #gluster
21:03 tessier Romanr: iperf shows the network is happy. On my currently unused cluster: [  3]  0.0-10.0 sec  1.09 GBytes   935 Mbits/sec
21:03 tessier On the one currently having trouble: [  3]  0.0-10.0 sec   295 MBytes   248 Mbits/sec
21:06 xoritor joined #gluster
21:07 Romanr change cable
21:07 Romanr or what this servers load is
21:08 xoritor ok i have used gluster successfully in the past, i am at a point now where i am using ceph quite well... but our current setup is about to under go a major change and may not support ceph in the future setup i am contemplating moving back to using gluster
21:08 JoeJulian Could, maybe, one of your bricks be read-only?
21:08 xoritor hi JoeJulian
21:08 JoeJulian tessier: ^
21:08 JoeJulian xoritor: cool
21:09 xoritor one of my major issues was high cpu usage on VM images... that was before libgfapi was widely adopted, and some other things were implemented
21:09 xoritor has that changed much?
21:10 xoritor has the cpu utilization gone down?
21:10 Romanr i'm running about 20 vm atm on pretty good gluster servs... load is 0.3
21:10 JoeJulian I couldn't really say since I've never had high cpu usage running VMs on gluster.
21:10 Romanr oh atm its even 0.01
21:11 xoritor i used to see my cpus pegged at 100% for hours on end
21:11 * Romanr thinks it could be only due to healing process o_O
21:11 xoritor yea
21:11 xoritor that was the issue
21:11 xoritor granted its been a few years
21:12 JoeJulian I see heals about once a quarter.
21:12 xoritor and JoeJulian helped me get some of it down with some settings i do not remember off the top of my head
21:12 xoritor ;-)
21:12 xoritor so thanks again JoeJulian
21:12 Romanr i bet it was disable self-heal :D
21:13 xoritor not sure
21:13 * Romanr is kidding
21:13 JoeJulian iirc we switched from delta to full heals
21:13 xoritor i think that was it
21:13 xoritor yea
21:13 JoeJulian Which will load network instead of cpu.
21:13 xoritor and that was fine
21:13 xoritor how is the rdma support with gluster?
21:14 JoeJulian Seems silly to me, but if the numbers make you happy...
21:14 xoritor i am looking at using it over rdma
21:14 tessier JoeJulian: I just checked. The xfs underlying my bricks are not read-only
21:14 JoeJulian I'd rather have 100% cpu usage and faster heals.
21:14 JoeJulian tessier: bummer. that would have been an easy fix.
21:14 tessier Indeed.
21:14 badone joined #gluster
21:14 xoritor yea until people cant do work in the other VMs running
21:14 tessier None of this makes any sense. :(
21:15 tessier We're supposed to be going live with this stuf now.
21:15 craigcabrey joined #gluster
21:16 tessier Tonight I'm going to reboot everything and test again
21:16 xoritor so one of the reasons i am looking into gluster again is native rdma support... the other is that ovirt supports it
21:16 xoritor anyone using both already?
21:16 xoritor ;-)
21:16 * xoritor thinks he is alone again
21:17 Romanr tessier, recreate everything, disable selinux everywhere, flush firewalls, switch cables with sftp / cat6 cables
21:17 Romanr i use only debian based systems. sorry
21:18 cleong joined #gluster
21:18 tessier Romanr: That's a lot of work for possibly zero gain. And given that I've already done that once (this is my second gluster setup) and they both have the same poor performance, I'm not optimistic that it will work.
21:18 Romanr then there must be something common between those setups. break it
21:19 * Romanr got himself a peppermint tea with honey and.. a coooooooookie!!!
21:19 Romanr tessier, actually i do not understand what could cause your issue. reproduce here step-by-step what u did and how.. or better send it to pastebin
21:19 xoritor here are the specs of the systems i have (12 cores, 128 GB DDR4, 1x256 GB SSD, 1x1 TB SSD, 1 2x40 Gbit IB)
21:20 tessier I'll pastebin the notes I used to set this up...
21:20 xoritor so if i can use rdma i will be happy
21:21 xoritor Romanr, got a cookie for me?
21:21 * Romanr thinks that gluster nor ceph is ready for such huge expectations when he sees 2x40Gbit cards in business.
21:21 Romanr xoritor, got lots of them.
21:22 * Romanr sends cookies to everyone. prepare your browsers :D
21:22 xoritor bwahahahahahahahaha
21:23 xoritor ceph is good storage... just not supported by what i am looking at using
21:24 xoritor its also a LOT harder to set up
21:24 xoritor and maintain
21:24 xoritor hah
21:26 Romanr and a lot slower
21:26 xoritor is it?
21:26 xoritor i get good results with it
21:26 Romanr with gluster those would be better
21:26 Romanr until u max out ur network
21:27 xoritor well i will max out the sata bandwidth first
21:28 Romanr https://fio.monaco.cx
21:28 Pupeno joined #gluster
21:29 xoritor where did you get my stuff?
21:29 xoritor ;-)
21:29 Romanr so thats ur link?
21:29 xoritor no
21:29 xoritor i actually have almost that exact setup
21:30 xoritor but i have 12 core xeons
21:30 xoritor 1 256 GB 850 pro
21:30 * Romanr thinks that there was some kind of retail sale and a lot of guys got such HW :D
21:30 xoritor 1 1TB 850 PRO
21:31 xoritor i do have the HGST 4 TB drives but they are not in the systems
21:31 Romanr well, at least you can compare performance.
21:31 Romanr gartner would charge 100500$ for such analytics. ..
21:31 xoritor lol
21:31 xoritor true
21:32 tessier Romanr: http://fpaste.org/245169/70823181/
21:32 xoritor this is AWESOME
21:32 xoritor thanks
21:32 Romanr np. someone shared this link here iirr
21:32 Romanr tessier, going to look at it
21:33 tessier It's really simple. I just basically followed the gluster quickstart tutorial.
21:36 Romanr for libgfapi there should be 1 key more added to volume and one to glusterd.vol, but i think you've made just not covered in file?
21:36 tessier Romanr: Correct, I did that.
21:37 JoeJulian tessier: Ok, going back to look at the EROFS *warning* (not error)... it's in the fuse_writev_cbk which can only be generated within gluster by the read-only, snapview-client, worm, nfs, or filter translators - none of which should apply unless you specifically set them to. That means the the error *must* be coming from the brick.
21:38 JoeJulian Oh, wait.. missed on line in my grep results...
21:39 Romanr but we should separate flies and steaks ... there are two problems atm: 1. setup is slow. 2. may be comes from 1st: VMs startu up with RO FS.. if speak about 2nd, its easy: such things may be due two factors: healing (damages virtual disk) and corrupted files (binaries) <- this one had I with my problem. Try SATA driver when 1st is solved and 2nd stays
21:40 Romanr so we should find out, why its slow
21:40 JoeJulian tessier: Ok, yeah, the only other way would be for quorum loss in which case there would be a line like, ""%s: Failing %s as quorum is not met"" preceeding the writev warning.
21:40 tessier JoeJulian: Yeah, I haven't set up any of that stuff nor do I know what that stuff is yet.
21:40 * tessier greps for quorum
21:41 tessier http://fpaste.org/245172/08287914/
21:41 tessier Lots of noise about quorum not being met
21:42 JoeJulian ~pasteinfo | tessier
21:42 glusterbot tessier: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
21:42 tessier Romanr: VMs only startup with RO FS if I am using libgfapi. WHich I am not at the moment.
21:42 tessier ok
21:42 * Romanr always advices to safe time and start over and over again, if quick tutors wont' work for some one. as there is some mistake made in setup likely 95%. so Lets listen what JoeJulian offers :))
21:43 JoeJulian It's stuff like this that makes me the expert. I actually read through the source to find the possibilities. :)
21:43 Romanr sure. you're dev, you ARE expert here :)
21:43 * PatNarciso nods.
21:43 Romanr even THE expert :P
21:44 * Romanr only advices some things from hes own experience.. and some expertise from networking (i'm network admin)
21:45 JoeJulian I'm not a dev.
21:45 JoeJulian I'm just a user like you guys.
21:45 * Romanr surprised
21:45 Romanr someone get an op status off him
21:45 Romanr :D
21:45 JoeJulian Hehe
21:45 xoritor no
21:45 tessier http://fpaste.org/245174/83138143/
21:45 xoritor he deserves it more than anyone i know
21:45 xoritor hahahaha
21:46 * xoritor does not really know JoeJulian
21:46 Romanr JoeJulian, but irl your are dev? not of gluster, but in general?
21:47 JoeJulian I hack at python. I contribute to salt a little.
21:47 xoritor just go to his blog it tells you all you want to know to stalk him
21:48 xoritor ;-)
21:48 xoritor just kidding JoeJulian
21:48 JoeJulian Yep
21:48 tessier Yep. Joe's got tons of good stuff on his blog.
21:48 tessier I've read pretty much all of the gluster stuff there
21:48 xoritor yep yep
21:48 Romanr well, at least you read code. I don't :)
21:48 JoeJulian I've got to get back to adding stuff.
21:48 Romanr so u r dev for me
21:48 * xoritor cracks the whip at JoeJulian --- get to writing
21:48 glusterbot xoritor: -'s karma is now -344
21:49 xoritor bwahahahahahaha
21:49 xoritor its all in good fun... but the bot cant take it
21:49 JoeJulian Nobody likes poor ol' -
21:49 xoritor hehehe
21:49 Romanr negative smells like negative.. call it negative or not ... (c)
21:50 xoritor - negative nelly
21:50 tessier That gluster volume info output looks pretty basic/normal to me.
21:51 xoritor i am seriously hungry for some thai food
21:51 xoritor masaman curry
21:52 tessier We have lots of good thai food around here.
21:52 Romanr What's in a name? that which we call a negative
21:52 Romanr By any other name would be as negative;
21:52 xoritor tessier, where are you?
21:52 Romanr from
21:53 Romanr :)
21:53 tessier xoritor: San Diego
21:53 tessier CA USA
21:53 xoritor oh yea... lots of good thai there
21:53 xoritor yea
21:53 xoritor i was stationed at pendelton
21:53 Romanr i like mexican food.
21:53 xoritor i like ALL food pretty much
21:53 Romanr tequila beer fajita, tacos
21:54 xoritor we have lots of that here in austin tx
21:54 Romanr you won't believe me, but we've also  got enough of it ... here in estonia :D
21:54 xoritor thats a bit strange to here
21:55 tessier 7234667 packets transmitted, 7195655 received, 0% packet loss, time 4536728ms
21:55 tessier I dunno, the layer 2 seems pretty solid.
21:55 xoritor but i had some decent mexican in helsinki
21:55 Romanr we even got a place named texas :D texas.ee (not and advertising )
21:55 xoritor s/here/hear/
21:55 glusterbot What xoritor meant to say was: An error has occurred and has been logged. Please contact this bot's administrator for more information.
21:56 JoeJulian dammit, glusterbot, why do you make me spend time fixing little shit that used to work?
21:56 tessier heh
21:56 xoritor s/my brain/abbey normal/
21:56 glusterbot What xoritor meant to say was: An error has occurred and has been logged. Please contact this bot's administrator for more information.
21:56 PatNarciso JoeJulian, what would ya recommend the setup of a 8x8tb 8gb server be.  1 brick on a large xfs over mdm?   8 separate bricks?   single server now, with intent to add nodes in upcoming months.
21:56 xoritor yes i broke it!!!
21:57 JoeJulian tessier: check your glusterd logs, /var/log/glusterfs/etc-glusterfs-glusterd.vol.log for clues.
21:57 JoeJulian PatNarciso: depends on the use case.
21:57 xoritor JoeJulian, so true
21:57 xoritor everything always comes down to that
21:57 PatNarciso true...
21:58 tessier JoeJulian: It contains lots of stuff like this: http://fpaste.org/245175/43708387/
21:58 PatNarciso lots of large video; redundancy is nice, but secondary.
21:58 xoritor PatNarciso, i would say try it one way and if it does not work you can back up the data and try it the other
21:58 tessier But I have no clue what that means. It mentions nfs. But I'm not running nfs anywhere here.
21:58 xoritor PatNarciso, you could even do 2x4
21:59 tessier NFS is one way to do it but fuse is simpler/gluster native, right? I would like to use the more efficient libgfapi but it doesn't seem to work as long as I have whatever other problem this is...
21:59 xoritor or 4x2
21:59 Romanr tessier, hmmm.. maybe there is a HW failure in ur server? memory tested?
21:59 xoritor so many options
21:59 PatNarciso xoritor, yeahhh... thats my problem.  I already got water in the buckets.  makes modifications... difficult.
22:00 xoritor PatNarciso, i usually like the 1x8 and control it from there just to make sure that if a brick fails it is easily replaceable
22:00 tessier Romanr: ECC memory. Hasn't had any other problems. You would think the OS etc. would be having trouble.
22:00 xoritor you dont have to do much redundancy... but just a little can save your butt
22:00 * Romanr usually likes 69
22:00 Romanr oops wrong place
22:00 Romanr :D
22:00 xoritor bwahahahahahahahaha
22:00 tessier JoeJulian: That last paste was from the client where the volume is mounted. Here's the noise in the server: http://fpaste.org/245177/37084021/
22:01 tessier [2015-07-16 13:54:54.759540] W [rpcsvc.c:270:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 10.0.1.15:1017
22:01 tessier [2015-07-16 13:54:54.759597] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
22:01 tessier It's hard to tell what is error, what is normal, and what any of it means. That bit there doesn't look good. What RPC program would it be looking for?
22:02 xoritor is portmapper installed?
22:02 xoritor i may just be trying to hit port 111
22:02 tessier http://fpaste.org/245178/43708413/
22:02 tessier Here's another interesting bit
22:02 tessier xoritor: Do I need portmapper?
22:02 JoeJulian ' E ' is an error.
22:02 tessier Ah, error, warning, information. Ok.
22:03 JoeJulian Where are you seeing that error? It's not in the last paste.
22:03 tessier Grepping for just E I get: http://fpaste.org/245180/37084193/
22:03 and` joined #gluster
22:03 PatNarciso xoritor, 1x8 - so, each physical drive would be its own brick. ?
22:03 xoritor yea
22:03 xoritor PatNarciso, thats just me though
22:04 PatNarciso I like to KISS.
22:04 tessier That's in the glusterd log from the server.
22:04 tessier Even though NFS hasn't been mentioned anywhere am I supposed to have it or portmap or something running?
22:04 PatNarciso xoritor: my concern with a previous, similar setup was... IO.
22:04 * Romanr likes to ... oops well. u know
22:04 tessier Hard to imagine it would work at all if that were the caes.
22:04 tessier case
22:05 JoeJulian If you could not filter those logs, that would be better. It's easier to tell how it got to the error with the rest of the log.
22:05 xoritor PatNarciso, thats the easiest thing for me to do... and i never had all of my drives in the same system
22:05 tessier JoeJulian: Ok, let me see if I can upload the whole log somehow...
22:05 xoritor Romanr, move it... move it?
22:05 Romanr xoritor, reset .. reset
22:05 tessier I'm in tmux so I can't easily cut and paste the whole log just by dragging the mouse up the termianl...
22:05 JoeJulian Which distro?
22:05 JoeJulian tessier: ?
22:05 JoeJulian @paste
22:05 glusterbot JoeJulian: For RPM based distros you can yum install fpaste, for debian, ubuntu, and arch it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
22:06 Romanr wow... some1 was really lazy
22:06 xoritor JoeJulian, for just a second there i thought you had those all memorized
22:06 tessier My brick servers don't have direct internet access.
22:06 tessier I can copy the log down locally to here and use fpaste though
22:07 PatNarciso I currently have a single server gluster setup on a 1x8 distributed volume.  IO was fine for about a month.  and then, well... it got slow.  (JoeJulian, how bad of an idea was this? )
22:07 JoeJulian Romanr: I live on lazy. Lazy is why I make the big bucks.
22:07 xoritor PatNarciso, was that due to a "heal" causing the slowdown?
22:08 PatNarciso xoritor, negative.
22:08 JoeJulian no, it's distribute, no replication. It cannot be heal.
22:08 tessier JoeJulian: http://paste.fedoraproject.org/245183/37084510
22:08 xoritor oh i missed that
22:08 * Romanr is thinking to become lazy now.
22:08 tessier That's the entire log, copied from the server to my workstation then uploaded with the fpaste app
22:09 xoritor Romanr, all admins are inherently lazy
22:09 Romanr I am not :)
22:09 xoritor it is why we are admins
22:09 xoritor you just have not accepted it yet
22:09 Romanr cuz i don't make big bucks :(
22:09 xoritor you will see in time...
22:09 PatNarciso the underlying bricks were formated xfs default.  I wonder how the gluster-suggested tweaks would have helped.
22:09 xoritor come to the dark side... the lazy side... we have cookies
22:10 xoritor wait... you have all the cookies!
22:10 tessier xoritor: Sometimes I'm lazy, sometimes I'm not. Right now I wish I could find a lazy way to solve this problem but it seems continuous hard work pounding on all possible options and reading everything I can find is my ony route.
22:10 Romanr i know your cookies addresses... brazzers.com ?
22:10 xoritor tessier, a little hard work from time to time allows us to be lazy the majority of the time
22:11 Romanr now i feel and sure i'm lazy.
22:11 Romanr thanx xoritor
22:11 xoritor we call it automation, orchestration, etc...
22:11 Romanr but still there is a problem.. where are my big bucks
22:12 tessier xoritor: I'm hoping that once I finally get gluster working properly it will make my life much easier compared to the way I used to do this using lvm->iscsi->mdadm
22:12 xoritor Romanr, i think JoeJulian got them
22:12 * Romanr has automated even automatic stuff :D
22:12 nage joined #gluster
22:12 xoritor tessier, it will... it is worth it it will work in the end, some things take tons of hard work to figure out
22:12 PatNarciso Romanr, that just blew my mind.
22:13 PatNarciso automatically.
22:13 Romanr see.. working
22:13 * Romanr automated it
22:13 tessier xoritor: The annoying thing about this is that it is surely one little thing I've done wrong somewhere...
22:13 JoeJulian tessier: did I ask for a "gluster volume status" yet?
22:14 Romanr but there are negative sides also.. I'm getting information about outages faster, than our noc.. they don't like me cuz of that.
22:14 JoeJulian tessier: That log looks fine during the timestamp of the earlier writev warning.
22:14 JoeJulian I wonder if there's anything on the other server.
22:14 tessier JoeJulian: Yes....but I'll give you another one...
22:14 tessier I'll check the other replicant in just a moment...
22:15 xoritor i am about to have to work very hard myself so i can go back to being lazy again
22:15 tessier JoeJulian: gluster volume status: http://fpaste.org/245186/37084944/
22:16 xoritor but before i do that... i think some masaman curry is in my future
22:16 Romanr you know, its like a call at 03:00 am, while i should be sleeping, but i pick up the phone and hear: Hi Roman! seems like there is an outage at @#@!@# ..damn. it is already not.. go to hell :D
22:16 xoritor speeeld et wrng
22:17 xoritor massaman curry
22:17 Romanr xoritor is bored. reading irc history
22:17 jmarley joined #gluster
22:17 tessier JoeJulian: http://paste.fedoraproject.org/245188/85066143 here is the glusterd log from the other replicant pair
22:18 xoritor actually i am waiting for all of the people to get off of the damned VMs so i can copy them off of rbd
22:18 xoritor then wipe all the ceph nodes and make them gluster bricks
22:18 _maserati_ I go up to Denver for the day and come back to all this? :)
22:18 Romanr haha! hi _maserati_
22:19 Romanr lucky u guys have day there.. 01:18 am here
22:19 tessier JoeJulian: My client machine seems to be 3 minutes behind the brick servers, if that matters.
22:19 xoritor _maserati_, do you have a maserati?
22:19 _maserati_ 3
22:19 Romanr wrong question
22:19 tessier I'll fix that...but gluster doesn't depend on having the times synched does it?
22:19 Romanr _maserati_, does your maserati has you ? :D
22:19 * xoritor like fast cars
22:19 Romanr does have*
22:20 _maserati_ Not sure who has who, but i prefer to be inside it
22:20 * Romanr prefers to be on top
22:20 xoritor i just bought a 2015 challenger scat pack
22:20 _maserati_ you're not car surfing on the maserati
22:20 xoritor not a maserati...
22:20 _maserati_ <3 Dodge, it's all good
22:21 Romanr ram 3500
22:21 JoeJulian Damn... I was hoping to find something around 2015-07-16 20:50:22
22:21 Romanr tuned
22:21 _maserati_ niiiice
22:21 JoeJulian Search for that timestamp in the brick logs.
22:21 _maserati_ my next ride is going to be a tuned 3500 (not a dually)
22:21 xoritor 485 horses is fun
22:21 xoritor thats before i start doing anything with it
22:21 _maserati_ srt8?
22:22 xoritor scat pack
22:22 JoeJulian tessier: Times should be synced, yes, but that shouldn't cause EROFS.
22:22 Romanr ok. lets stay quite for a while. we bother joe and tess
22:23 JoeJulian tessier: that timestamp I said to look for was the time (in UTC) from the client. If the clocks are off, adjust accordingly.
22:23 * Romanr is away: Really :-)
22:23 tessier JoeJulian: http://fpaste.org/245191/14370853/ That's the closest I could find.
22:24 JoeJulian tessier: brick logs are under /var/log/glusterfs/bricks
22:24 tessier oh, ok
22:25 tessier JoeJulian: http://fpaste.org/245192/43708551/
22:26 tessier JoeJulian: Exact same thing on both servers, aside from a second difference in time
22:26 tessier That doesn't look like errors though
22:36 JoeJulian tessier: it clearly shows the clients disconnecting and connecting which must be why you're getting read only status. I can't see any reason for that though.
22:36 JoeJulian I'd probably wireshark this next.
22:36 JoeJulian But that's a lot of work for a PoC.
22:36 tessier I've already got a packet capture but I don't know if it has anything interesting in it...
22:36 Romanr joined #gluster
22:36 JoeJulian Try 3.6 instead of 3.7 and see if it works any differently.
22:41 Romeor ok. I've now registered that nick name to me. So if its possible I'd like to have an auto +v :)
22:42 * tessier just captured around 100MB worth of traffic
22:42 tessier Hmm...more like a couple gig of traffic.
22:43 Romeor good night every1
22:44 * Romeor is away: Really :-)
22:46 tessier Tons of "TCP Dup ACK" in here.
22:49 tessier Lots of Dup ACK and retransmissions.
22:49 tessier But then why did iperf report everything ok? And ping works without dropping packets?
22:49 gildub joined #gluster
22:50 tessier Tonight I'm going to shutdown everything, replace the switch and all cabling...and if that doesn't fix it tomorrow management is going to force me to give up on this and go back to iscsi which we know works even though it's a pain to work with. :|
22:51 Romeor tessier, thats the point of tcp. retransmissions...
22:51 Romeor try iperf with udp
22:51 tessier Romeor: There shouldn't be retransmissions with a properly working network. Especially not this many.
22:52 Romeor agree. but there is an answer in you line to ur question
22:52 tessier ?
22:52 Romeor seems like not properly working network?
22:53 JoeJulian could be packet size
22:53 Romeor check arp logs, mby you've got multiple hosts with same ips?
22:53 JoeJulian +1
22:53 tessier hrmmm
22:54 tessier That would be interesting.
22:54 Romeor or some angry guy arp poisoning you if it is possible
22:55 * Romeor likes trolling office workers with arp poisoning, changing any <html img src=....> with my own :)
22:55 natarej_ does gluster have some sort of write failsafe to prevent corruption?
22:55 tessier It's a private network behind two firewalls internet->dmz->SAN/private...so I would hope that's not possible. But I'm looking into it...
22:55 natarej_ eg using the disperse translator and you lose 2/3 nodes.
22:56 natarej_ will clients continue to attempt to write to the single remaining node?
22:57 JoeJulian natarej_: I would think not. A 2/3 loss of EC data is a full data loss.
22:57 side_control joined #gluster
22:59 side_con1rol joined #gluster
23:00 Romeor what should i do to get auto voice ? 8)
23:01 natarej_ JoeJulian, so there is no failsafe / read-only mode?
23:01 JoeJulian natarej_: there couldn't possibly be. With only 1/3 the data, there is no data.
23:02 natarej_ i feel like you're misunderstanding
23:03 natarej_ lets say we have a power outage.  3/3 nodes down.  when everything comes back up i assume you would run some sort of heal and you've got a pretty good chance of having data integrity
23:04 natarej_ however if you have 2/3 nodes down, one remains active - if it detects the other nodes are missing and doesn't allow files to be written, you have the same chance of having data integrity when the other two nodes come back up.
23:05 JoeJulian Sure, but you still couldn't *read* data because it's just not there and can't be assembled.
23:05 tessier Hmm...if I had an IP or MAC conflict somewhere it would affect my ability to ssh into the boxes. And it hasn't...
23:05 JoeJulian tessier: maybe, but not always.
23:05 Romeor +1
23:05 natarej_ the point is it wouldn't attempt to write to the cluster
23:06 Romeor tessier, the most easy way to look for arp changes is to look at router logs.
23:06 xoritor arpwatch
23:06 * tessier just installed arpwatch
23:06 xoritor lol
23:07 Romeor or tcpdump for arp
23:07 tessier hmm...
23:07 natarej_ in that event, when the other two nodes come back up they would be considered the ones with data integrity and the one that remained up is guaranteed to have corrupt data, significantly increasing the chance to total data loss
23:07 tessier I have an interface em1:
23:07 tessier em1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 9000 ether d4:be:d9:ef:d7:66  txqueuelen 1000  (Ethernet)
23:08 tessier and an interface em2:
23:08 tessier em2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 9000 ether d4:be:d9:ef:d7:66  txqueuelen 1000  (Ethernet)
23:08 Romeor o_O freebsd?
23:08 xoritor if you have elasticsearch you can add packetbeat and feed it
23:08 xoritor em1 embeded interface1
23:08 xoritor wow same mac
23:08 JoeJulian natarej_: I'm just saying it can neither write nor read during that proposed outage. It cannot recover until at least one other server has recovered.
23:08 tessier They are LACP bonded...
23:08 tessier That's what ifconfig says for their macs.
23:09 tessier But cat /proc/net/bonding/bond0  says:
23:09 tessier Permanent HW addr: d4:be:d9:ef:d7:66
23:09 tessier Permanent HW addr: d4:be:d9:ef:d7:68
23:09 tessier I think that's an LACP thing...
23:09 * tessier checks another machine
23:10 tessier Yeah...that's normal. :(
23:10 tessier I had hoped I had found something broken. But that's normal with LACP
23:11 Romeor soooo. may be there is a wrong config between linux and switch?
23:11 Romeor wrong way bonding?
23:11 Romeor try without bonding
23:11 xoritor they should not have the same mac on the em1 and em2 the mac should be set on the bond
23:11 * JoeJulian pokes semiosis ... bonding again...
23:11 Romeor and xoritor is right
23:11 natarej_ JoeJulian, it's not about recovery
23:11 xoritor i will bet you the bond is the issue
23:11 xoritor ok
23:11 xoritor but i would try it without bonding first and see what happens
23:11 Romeor tessier, just remove bonding and try. if everything will be ok, it'll be simple to track the mistake down
23:11 Romeor exclusive method rocks
23:11 xoritor it sounds to me like it is getting an arp due to nic failover at the switch
23:11 Romeor it could be normal if switch (on the other side) supported it
23:12 xoritor make sure your switch is using the right lacp methods
23:12 Romeor and balancing
23:12 xoritor well... the same methods that your server is using
23:12 xoritor ie... consistancy
23:12 xoritor spelling?
23:12 Romeor but the easiest thing is to get rid of bond
23:12 Romeor its not that hard
23:13 xoritor and turn it off on the switch so it is not looking for it on your server
23:13 JoeJulian natarej_: One server down and no quorum, reads and writes may continue. Two servers down, no reads nor writes. Three servers down, no reads nor writes. One server comes back, no reads nor writes. Two server come back and no quorum restriction, reads and writes possible. Three servers come back, the third server is stale and is healed.
23:14 xoritor and if you have a bond with lacp make sure it is not trying to issue stp also
23:14 Romeor and see if it is active mode on switch side
23:14 xoritor Romeor, i am leaving its all you
23:14 xoritor finally getting my curry
23:17 Romeor there is nothing to leave, but thnx :D /me is going to sleep also. seems to me like wrong bonding. never seen same macs on interfaces.
23:21 natarej_ JoeJulian: the question is not whether the client can successfully complete a read or write while the cluster has failed...
23:21 Romeor but mby i configured it other way
23:21 natarej_ i'll just test it myself
23:21 xoritor Romeor, it can be reported back by some tools that way
23:21 xoritor depends on the methods used... i typically do not use linux bonding anymore opting for ovs or some other means
23:21 Romeor well, if its RR then same macs ok
23:21 Romeor tessier, are u balancing? what method is used?
23:21 xoritor unless the switch does not support that
23:21 xoritor ;-)
23:21 crashmag_ joined #gluster
23:22 Romeor should work is its is rr and active lacp on switch side
23:23 Romeor but even then i would remove bonding and try
23:23 xoritor yea
23:24 xoritor that is the easiest test
23:25 natarej_ JoeJulian, I found my answer.  The failsafe does exists in the form of cluster.server-quorum-ratio introduced in 3.4.  thank you anyway.
23:26 Romeor shower
23:31 tessier Romeor: I've set up LACP loads of times before. This is set up the exact same way as all of the others.
23:32 tessier Romeor: options bond0 miimon=100 mode=4 lacp_rate=1
23:33 tessier 11   | cpu05                            100/1000T | Trk4  LACP
23:33 tessier And that's from the switch. That's all you need.
23:33 tessier I'll try removing the bonding as I simplify the network situation but I don't see any reason why that would help.
23:44 tessier I should be able to connect a brick directly to the client, right?
23:44 tessier No switch.
23:45 tessier That would be the simplest possible network configuration.
23:45 tessier But without being able to transfer a packet capture to my laptop to run wireshark I'm not sure how I would really tell what is happening on the wire.
23:49 calavera joined #gluster
23:49 PatNarciso joined #gluster
23:53 calavera joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary