Camelia, the Perl 6 bug

IRC log for #gluster, 2012-10-19

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:22 bjoernt left #gluster
00:43 y4m4 joined #gluster
01:14 Bullardo joined #gluster
01:26 ngoswami joined #gluster
01:34 lng joined #gluster
01:36 lng Hi! IP addresses of 2 nodes changed and I'm trying to reconfigure the storage. I tried to add peer with the new address, but having 'Probe on host xx.xx.xx.xx port 0 already in peer list'. What is the solution?
01:42 lng Now I'm trying namebased setup but having most likely the same error: 'Probe on host hostname.com. port 0 already in peer list'
01:42 lng Shall I remove peer first?
01:43 JoeJulian You won't be able to remove a peer that's part of a volume.
01:45 JoeJulian This is why we always stress to use hostnames when creating your volumes.
01:46 JoeJulian If the UUID is intact in /var/lib/glusterd/glusterd.info on the machines whose addresses changed, you should be able to convert them to hostnames by probing them from the good peer.
01:46 JoeJulian s/.$/by name./
01:46 glusterbot What JoeJulian meant to say was: If the UUID is intact in /var/lib/glusterd/glusterd.info on the machines whose addresses changed, you should be able to convert them to hostnames by probing them from the good peerby name.
01:46 lng JoeJulian: this is test setup
01:46 lng two nodes / two bricks
01:46 lng both addresses changed
01:46 JoeJulian Ah, ok then. Just wipe /var/lib/glusterd and start over. :)
01:47 JoeJulian use hostnames.
01:47 lng great!
01:47 JoeJulian And, in case I came across as indecisive, use hostnames. ;)
01:47 lng JoeJulian: while you are here, I have one more question
01:47 JoeJulian I'm running away... :P
01:47 lng related
01:47 lng pls
01:47 lng 1 min
01:47 JoeJulian just kidding
01:47 lng yesterday
01:48 lng I added peers over
01:48 lng before  - IP
01:48 lng now I added hostnames
01:48 lng IPs _not_ changed
01:48 JoeJulian One of my pet peeves is fragmented sentences. Not to say you're doing it wrong, but feel free to complete your thought before pressing enter. I'll still be here.
01:49 lng but when I grep /var/lib/glusterd for IPs, they are still there
01:50 JoeJulian Any volumes you created using those ip addresses will remain ip addresses. Changing the peer to a hostname won't undo that.
01:50 JoeJulian I've entered an enhancement request related to that issue. Probably not going to make 3.4
01:51 lng JoeJulian: the reason I turned to hostnames is that I need to stop/strart one of the nodes, but after that IP will change
01:52 lng JoeJulian: what if I replace all the IPs in /var/lib/glusterd to new hostnames?
01:52 JoeJulian It /should/ work as long as glusterd and the volumes are stopped.
01:53 lng JoeJulian: what do you mean?
01:53 lng JoeJulian: no need to replace IPs?
01:55 JoeJulian stop your volume(s), "gluster volume stop $volname". service stop glusterd (or your equivalent if you're not using RHEL/CentOS). Replace those IPs with hostnames. Be sure the hostnames resolve. service glusterd start. gluster volume start $volume
01:56 JoeJulian I'm 95% sure that will work.
01:56 lng JoeJulian: I think I can replace IPs before I stop it
01:56 lng then restart
01:57 JoeJulian Let me know if that works for you.
01:57 lng but it's not good - downtime
01:58 lng I thought I can reconfigure it one by one
01:58 lng without stopping the vol.
01:58 JoeJulian So this isn't the test volume you mentioned a moment ago then.
01:58 lng yea, it's live
01:58 phansold joined #gluster
01:58 JoeJulian Then you're hosed.
01:58 lng no
01:59 lng I told you about another question
01:59 JoeJulian You're going to have downtime. Use something like cssh and you should be able to complete the change in about 10 seconds.
02:00 JoeJulian brb, my daughter wants my help with something.
02:00 lng sure
02:00 lng Cluster SSH?
02:01 lng never tried that...
02:02 phansold Hi there! I have an application that runs on 2 VMs and needs to write and access files to/from shared storage. I currently use RHEL 6 storage cluster (GFS2) on VMs (vSphere 4.1) with RDMs. This involves a few daemons (cman, rgmanager, qdiskd, clvmd), fencing, etc. Would GlusterFS be able to replace my RHEL 6 storage cluster and provide the same functionality?
02:04 JoeJulian lng: cssh is awesome for when you need to do the same command on multiple systems simultaneously.
02:04 lng JoeJulian: yea
02:04 JoeJulian phansold: I don't see why not.
02:05 phansold ok, I've just started reading GlusterFS docs...so, I'll have to install the servers and clients on both VMs to do this, is it not?
02:06 lng JoeJulian: I have noticed when Gluster mount is not available, Clients are very slow
02:06 lng JoeJulian: is it common problem?
02:06 JoeJulian Not usually. Are you using tiny instances or something?
02:06 lng JoeJulian: small one
02:06 lng ones*
02:07 lng for Gluster nodes, and medium for app clients
02:07 JoeJulian Next time he's around, ask semiosis what he recommends.
02:08 lng JoeJulian: I still have 12 days to reconfigure it
02:08 JoeJulian He's in GMT-4 and is usually around during normal business hours.
02:09 lng Canada?
02:09 JoeJulian Florida
02:10 lng wow - 12 hours difference
02:10 phansold JoeJulian: Thank you for the reply. RHEL 6 storage cluster uses fencing to protect against data corruption. GlusterFS architecture is different, but if one of the 2 nodes running GlusterFS (server/client) crashes, are there any risks of data corruption or just transient data loss?
02:12 JoeJulian The only normal potential for data loss from what your describing would be if there was a network partition and both clients continued updating files on their locally visible server. Your software would also have to do all the normally appropriate locking as well.
02:12 lng JoeJulian: is it the only directory where I should change IP addresses, /var/lib/glusterd?
02:14 sunus joined #gluster
02:15 lng JoeJulian: is there a way to start volume from ebs snapshots so that data will be present on a new volume?
02:17 JoeJulian I've never used AWS, so I don't know. That's why I was suggesting semiosis. He does use AWS quite extensively.
02:17 lng JoeJulian: I see, I see... Thanks!
02:17 JoeJulian And yes, /var/lib/glusterd/* is where you change the ips
02:18 lng JoeJulian: thank you for your help!
02:18 JoeJulian You're welcome.
02:21 phansold JoeJulian: Thanks.
02:22 JoeJulian phansold: You're welcome.
02:36 lng JoeJulian: for cssh, how can use rsa key?
02:37 JoeJulian just like ssh
02:50 sgowda joined #gluster
03:02 lng 'cssh -i ~/pem/key.pem ubuntu@host' - not working
03:03 lng I see no -i switch
03:03 lng is it clusterssh package?
03:06 lng I might need to configure it
03:08 bharata joined #gluster
03:23 zwu joined #gluster
03:28 hagarth joined #gluster
03:30 shylesh joined #gluster
03:53 hagarth joined #gluster
03:59 bulde joined #gluster
04:14 glusterbot New news from newglusterbugs: [Bug 868087] Multiple memcache key/value pairs are used to store metadata for account and containers <https://bugzilla.redhat.com/show_bug.cgi?id=868087> || [Bug 868086] Fetching the object metadata involves multiple stat() system calls when one suffices <https://bugzilla.redhat.com/show_bug.cgi?id=868086>
04:20 vpshastry joined #gluster
04:22 JoeJulian joined #gluster
04:22 sgowda joined #gluster
04:33 bulde joined #gluster
04:41 crashmag joined #gluster
04:48 lng semiosis: Hello! I have question about Gluster reconfiguration. I need to change to namebased setup... It is live and it's on EC2. I have added peers with host names already, but still see the IP addresses in /var/lib/glusterd/. What should I do next?
04:48 lng (before it was configured by IPs)
04:49 lng EC2 schedulet Instance stop event and I have 12 days to resolve the situation
04:56 deepakcs joined #gluster
04:58 bulde joined #gluster
05:02 faizan joined #gluster
05:05 bulde joined #gluster
05:16 bulde joined #gluster
05:22 bulde joined #gluster
05:26 stat1x joined #gluster
05:27 ramkrsna joined #gluster
05:27 ramkrsna joined #gluster
05:37 sac_ joined #gluster
05:44 crashmag joined #gluster
05:44 glusterbot New news from newglusterbugs: [Bug 857220] [enhancement]: Add "system restart" command for managing version upgrades <https://bugzilla.redhat.com/show_bug.cgi?id=857220> || [Bug 858275] Gluster volume status doesn't show disconnected peers <https://bugzilla.redhat.com/show_bug.cgi?id=858275> || [Bug 864499] gluster volume geo-replication start --xml outputs wrong xml structure <https://bugzilla.redhat.com/show_bug.cgi?id=8
05:54 ngoswami joined #gluster
05:55 bala joined #gluster
05:57 sgowda joined #gluster
06:00 rosco__ joined #gluster
06:05 raghu joined #gluster
06:08 overclk joined #gluster
06:10 guigui3 joined #gluster
06:11 ngoswami joined #gluster
06:12 faizan joined #gluster
06:29 rgustafs joined #gluster
06:33 lng JoeJulian: maybe I can reboot only 1 node... at least I hope so
06:35 kevein joined #gluster
06:37 puebele1 joined #gluster
06:43 ctria joined #gluster
06:46 faizan joined #gluster
06:53 ika2810 joined #gluster
06:56 puebele joined #gluster
06:58 Azrael808 joined #gluster
07:01 andreask joined #gluster
07:09 Nr18 joined #gluster
07:14 tjikkun_work joined #gluster
07:14 ekuric joined #gluster
07:18 TheHaven joined #gluster
07:23 dobber joined #gluster
07:23 quillo joined #gluster
07:27 lkoranda joined #gluster
07:28 lkoranda joined #gluster
07:54 faizan joined #gluster
07:56 gbrand_ joined #gluster
08:09 manik joined #gluster
08:47 saz joined #gluster
08:50 sunus joined #gluster
08:50 lkoranda joined #gluster
08:51 mdarade joined #gluster
08:53 TheHaven joined #gluster
08:55 lng anybody use UCARP?
08:58 mdarade3 joined #gluster
08:58 Humble joined #gluster
09:19 samu60 joined #gluster
09:29 samu60 hi all
09:29 samu60 I've recently got hands on a Infiniband hardware (2 40G cards and a switch from Mellanox)
09:29 samu60 and I'm fighting against dependencies and performance
09:30 samu60 I've read that 3.3 does not support rdma, and so far I could ony make it work with version 3.2.7
09:31 samu60 but performance is horrible (10MB/s on write)
09:31 samu60 I'm using Centos 6.3 and OFED 1.5.4
09:31 samu60 is there anyone around with experience on this that can help a little?
09:31 samu60 thanks in advance
09:32 flowouffff where did u read that RDMA is not available for 3.3 ?
09:33 flowouffff you're right
09:33 flowouffff http://community.gluster.org/q/how-i-can-tro​ubleshoot-rdma-performance-issues-in-3-3-0/
09:33 glusterbot Title: Question: How I can troubleshoot RDMA performance issues in 3.3.0? (at community.gluster.org)
09:34 saz joined #gluster
09:34 samppah samu60: i don't have experience on that but many people have said that they have got better performance with IPoIB
09:35 flowouffff i've never use IPoIB so far
09:35 flowouffff i can't really tell
09:35 flowouffff :)
09:35 flowouffff maybe you should post on a the forum
09:36 samu60 I'm on it, but I'm trying to get a broad view of the issue
09:37 samu60 ;)
09:38 flowouffff have u tried the tweak some option like mentionned in the article
09:38 samu60 I'll try now
09:39 samu60 but I just wanted to know whether investing time on rdma on 3.3 or even 3.4-rc1
09:39 samu60 that's the main point of asking here....
09:43 flowouffff how fast can u Read with ur current configuration ?
09:43 flowouffff what kind of setup do u have?
09:43 flowouffff replicate, distributed, replicate distributed?
09:44 flowouffff I guess that some setup might be better for read or writes performance :)
09:44 flowouffff +s
09:44 flowouffff sorry, my english is a bit rusty these days
09:45 samu60 I have one single server without any special configuration
09:45 samu60 I got 10MB/s
09:45 flowouffff what kind of disk spec do u have ?
09:45 samu60 without any performance change
09:48 bala joined #gluster
09:48 samu60 I'll post a dd on the server
09:48 samu60 on the local disk
09:50 samu60 dd if=/dev/zero of=testInfi bs=1024 count=1M
09:50 samu60 377 MB/s
09:50 samu60 that's the local disk figures
09:50 faizan joined #gluster
09:51 andreask that includes writing to the page-cache
09:51 samu60 dd if=/dev/zero of=testInfi bs=1024 count=1M
09:52 samu60 11.8 MB/s
09:52 samu60 on gluster on the same disk
09:52 flowouffff nice perf locally
09:52 flowouffff once u go through the volume, u get 10mb?
09:52 samu60 using 3.2.7
09:52 samu60 yeah
09:52 samu60 on the client, using the mounted gluster volume, I got only 10MB
09:53 samu60 I can tune volume, but it's an "extrange" low value.....
09:53 samu60 gluster> volume info
09:53 samu60 Volume Name: test
09:53 samu60 Type: Distribute
09:53 samu60 Status: Started
09:53 samu60 Number of Bricks: 1
09:53 samu60 Transport-type: tcp,rdma
09:53 samu60 Bricks:
09:53 samu60 Brick1: nodo:/mnt/second
09:53 samu60 I've got it on rdma and tcp, but using only rdma has the same value
09:54 flowouffff how do u mount the volume ?
09:54 flowouffff what command do u use ?
09:55 samu60 I've got it now on TCP
09:55 samu60 mount -t glusterfs nodo:/test -o transport=tcp /mnt/testgluster
09:55 samu60 but I've also tried
09:55 samu60 mount -t glusterfs nodo:/test -o transport=rdma /mnt/testgluster
09:55 samu60 to use RDMS
09:55 samu60 RDMA
09:55 flowouffff same results?
09:55 samu60 around 10MB/s on both scenarios
09:56 flowouffff try to specify a log file with -o log-file
09:56 flowouffff and see if everything is well set up on the client side
09:57 flowouffff what command do u used to create ur volume ?
09:58 mdarade1 joined #gluster
09:58 flowouffff what command did u use to create ur volume ?
09:58 samu60 volume create test transport rdma nodo:/mnt/second
09:59 samu60 applied recomendations from communitiy.gluster.org:
09:59 samu60 performance.io-thread-count: 32
09:59 samu60 performance.write-behind-window-size: 32MB
09:59 samu60 performance.cache-size: 1024MB
09:59 samu60 umount client
09:59 samu60 and mount again with rdma
09:59 samu60 I'll perform dd again
09:59 sunus joined #gluster
09:59 samu60 and I'll look at the logs....one second
10:01 samu60 dd if=/dev/zero of=testInfi bs=1024 count=1M
10:01 samu60 10.7 MB/s
10:01 flowouffff those perfs look like 100MB/s ethernet speed
10:01 flowouffff that's wierd
10:02 flowouffff grep -iR rdma /var/log/glusterfs ?
10:03 samu60 it has failback to tcp
10:04 flowouffff ah
10:04 samu60 I'll set the volume with rdma only and come back
10:04 samu60 but the values are the same...I guess
10:04 samu60 give me a minute
10:04 flowouffff nothing in logs about RDMA issue ?
10:04 samu60 and thanks a lot
10:05 samu60 there are logs with rdma, but from older tests and it's hard to map to the scenario used
10:05 samu60 I'll create again
10:06 flowouffff u can clean ur logs folder
10:07 vpshastry joined #gluster
10:07 flowouffff recreate ur volume with RDMA support
10:07 samu60 gluster> volume create test transport rdma nodo:/mnt/second
10:07 samu60 Creation of volume test has been successful. Please start the volume to access data.
10:07 samu60 gluster> volume info
10:07 samu60 Volume Name: test
10:07 samu60 Type: Distribute
10:07 samu60 Status: Created
10:07 samu60 Number of Bricks: 1
10:07 samu60 Transport-type: rdma
10:07 flowouffff then start over ur tests
10:07 samu60 Bricks:
10:07 samu60 Brick1: nodo:/mnt/second
10:07 flowouffff ok start the volume now
10:07 samu60 volume start test
10:07 samu60 Starting volume test has been successful
10:07 samu60 mount -t glusterfs nodo:/test -o transport=rdma /mnt/testgluster
10:07 samu60 on client
10:07 flowouffff yes
10:08 samu60 012-10-19 12:07:47.842597] E [rdma.c:4417:tcp_connect_finish] 0-test-client-0: tcp connect to 192.168.1.2:24008 failed (Connection refused)
10:08 samu60 [2012-10-19 12:07:50.846857] E [rdma.c:4417:tcp_connect_finish] 0-test-client-0: tcp connect to 192.168.1.2:24008 failed (Connection refused)
10:08 samu60 [2012-10-19 12:07:53.851105] E [rdma.c:4417:tcp_connect_finish] 0-test-client-0: tcp connect to 192.168.1.2:24008 failed (Connection refused)
10:08 samu60 [2012-10-19 12:07:56.855344] E [rdma.c:4417:tcp_connect_finish] 0-test-client-0: tcp connect to 192.168.1.2:24008 failed (Connection refused)
10:08 samu60 mmmm
10:08 flowouffff sounds like he's trying to force transport TCP
10:08 samu60 that means that is falling back to TCP internally?
10:09 flowouffff maybe
10:09 flowouffff but
10:09 flowouffff you may have a problem with RDMA on the client
10:09 flowouffff try to grep -iR rdma /var/log/glusterfs ?
10:09 flowouffff on both side, client and server
10:10 samu60 2012-10-19 12:08:27.867116] I [glusterd-handler.c:1079:glus​terd_handle_cli_start_volume] 0-glusterd: Received start vol reqfor volume test
10:10 samu60 [2012-10-19 12:08:27.867171] I [glusterd-utils.c:243:glusterd_lock] 0-glusterd: Cluster lock held by b581ea47-8df9-4411-b6ff-687968b76433
10:10 samu60 [2012-10-19 12:08:27.867195] I [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local lock
10:10 samu60 [2012-10-19 12:08:27.867436] I [glusterd-op-sm.c:6737:glu​sterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 0 peers
10:10 samu60 [2012-10-19 12:08:27.867548] I [glusterd-utils.c:1097:glust​erd_volume_start_glusterfs] 0-: About to start glusterfs for brick nodo:/mnt/second
10:10 samu60 [2012-10-19 12:08:27.944932] I [glusterd-op-sm.c:6854:glus​terd_op_ac_send_commit_op] 0-glusterd: Sent op req to 0 peers
10:10 samu60 [2012-10-19 12:08:27.945028] I [glusterd-op-sm.c:7250:glusterd_op_txn_complete] 0-glusterd: Cleared local lock
10:10 samu60 [2012-10-19 12:08:28.571475] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /mnt/second on port 24012
10:10 samu60 [2012-10-19 12:08:28.573229] E [rdma.c:4468:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 192.168.1.2:1021) after handshake is complete
10:10 samu60 [2012-10-19 12:08:48.243648] E [rdma.c:4468:rdma_event_handler] 0-rpc-transport/rdma: rdma.management: pollin received on tcp socket (peer: 192.168.1.3:1022) after handshake is complete
10:10 samu60 [2012-10-19 12:10:45.477005] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (127.0.0.1:1023)
10:11 glusterbot samu60: That's just a spurious message which can be safely ignored.
10:13 samu60 so can I assure it's using RDMA?
10:14 samu60 if i use network sniffer tools and see nothing, can I assure it's using RDMA instead of TCP?
10:14 faizan joined #gluster
10:16 flowouffff i dont get why it's using TCP socket to complete the handshake
10:16 flowouffff you should contact the dev team to get your answers
10:16 samu60 the spurious message that glusterbot said is about the transport erroer?
10:16 samu60 which dev team?
10:17 samu60 gluster's? mellanox's?
10:17 flowouffff gluster's dev team :)
10:17 flowouffff maybe  you should post something on their forum
10:17 samu60 I hope anyone around here would be from the dev team
10:17 samu60 ;)
10:19 flowouffff they must be busy right now :)
10:19 samu60 thanks a lot for your attention
10:20 hagarth joined #gluster
10:20 bharata joined #gluster
10:20 mgebbe joined #gluster
10:20 flowouffff you're welcome
10:20 ekuric joined #gluster
10:52 edward1 joined #gluster
10:53 ekuric joined #gluster
10:56 zoldar I've put some mailboxes for testing on a gluster volume, which are accessed with courier-imap. The problem is, that all operations are slow as hell. Several seconds pass until any operations completes. I've observed that when it happens, following entries appear in volume's logs: http://dpaste.com/815494/ . From what I know Gluster 3.3 shouldn't require any additional setup to handle dotlock locking. What may be wrong?
10:56 glusterbot Title: dpaste: #815494 (at dpaste.com)
11:11 lng semiosis: are you here?
11:13 ika2810 left #gluster
11:14 Humble joined #gluster
11:15 vpshastry left #gluster
11:16 manik joined #gluster
11:20 kkeithley joined #gluster
11:42 TheHaven joined #gluster
11:45 vpshastry joined #gluster
11:45 vpshastry left #gluster
11:53 plarsen joined #gluster
12:01 puebele3 joined #gluster
12:07 balunasj joined #gluster
12:12 puebele joined #gluster
12:13 mdarade1 joined #gluster
12:31 saz joined #gluster
12:32 puebele1 joined #gluster
12:46 sashko joined #gluster
12:47 vipkilla joined #gluster
12:47 hchiramm_ joined #gluster
12:47 vipkilla anybody here that remembers my issue. i have resolved it without messing with kernel modules
13:00 JoeJulian vipkilla: how'd you do it?
13:00 vipkilla gfs01:/datastore /mnt/datastore glusterfs noatime,nodiratime,defaults,_netdev 0 0
13:00 vipkilla in fstab....
13:00 vipkilla someone should document that somewhere....
13:02 bala joined #gluster
13:03 samppah vipkilla: _netdev is the key to the success?
13:06 vipkilla samppah: i'm not sure that was it because i tried that yesterday
13:06 vipkilla whatever it is, that is the magical fstab for debian...
13:17 JoeJulian zoldar: xtime extended attributes are associated with geo-replication. Are you using that?
13:18 zoldar JoeJulian: yes
13:20 mdarade1 left #gluster
13:20 zoldar JoeJulian: that brings me back to my previous question. I have a geo-replication process which seems to work, has "OK" status. But doesn't react to neither "stop" (session not active) nor "start" (session already started). How to best clean it up?
13:22 JoeJulian maybe try restarting glusterd?
13:22 * JoeJulian is suggesting that way too often.
13:23 zoldar JoeJulian: I did a complete restart but after starting the process initiates automatically and behaves the same
13:24 JoeJulian I wonder if you can specify excludes for geo-rep... Might take care of the dotlock file issue...
13:27 JoeJulian hmm, nope.
13:27 JoeJulian someone should file a bug asking for that enhancement
13:27 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
13:30 13WABN59V joined #gluster
13:34 Nr18 joined #gluster
13:35 zoldar now I've disabled the node that did geo-replication but the same log messages still appear - so it doesn't seem to be xtime related... or does it?
13:36 zoldar by node I meant a node which was geo-replication master, not slave - there's a second node serving the volume
13:38 JoeJulian The xtime markers are part of the volume definition, so stopping one server won't stop it from trying to create those.
13:39 JoeJulian Is there a way to use a different directory path for the lock files (or, even better would be memcached)?
13:41 JoeJulian Cool. That was fairly painless. I just migrated 30 bricks to their new server. Half way done.
13:41 zoldar I would have to look into how accomplish that with courier
13:42 * jdarcy o_O
13:42 JoeJulian memcached would be what I would do if it's possible. Otherwise if you can do a different directory tree, you could create a non-geocached volume just for the shared locks.
13:44 zoldar JoeJulian: If I wanted to make sure that geo-replication was the cause, how do I remove those markers? Preferably without tearing down the whole volume?
13:46 glusterbot New news from newglusterbugs: [Bug 868314] replace-brick should be able to continue <https://bugzilla.redhat.com/show_bug.cgi?id=868314>
13:46 JoeJulian zoldar: As an aside, are you stuck on courier? For putting imap on gluster, I would use dovecot's maildir-mbox hybrid (I believe it's called dbox). maildir's going to suck because of self-heal checks (jdarcy's working on making it suck less) and, of course, mbox just sucks in general. The hybrid seems to minimize the suckage.
13:47 johnmark +1
13:48 zoldar JoeJulian: I sure will migrate to dovecot, but for a transitional period I need a working courier
13:49 JoeJulian Code obfuscation made my answer take longer.... :(
13:49 JoeJulian You can disable marker with "gluster volume set $vol geo-replication.indexing off"
13:50 zoldar I thought dsync would be a nice alternative but from what I see it's not ready yet for full production usage and Timo is in the middle of rewriting it
13:50 zoldar JoeJulian: hmm right, but for the change to take effect the geo-replication has to be stopped, right?
13:51 JoeJulian I wouldn't expect it to need that.
13:51 zoldar ok, will give it a try in a moment
13:52 Nr18 joined #gluster
13:54 JoeJulian This makes reading code more fun: {VKEY_MARKER_XTIME,"features/marker","xtime", "off", NO_DOC, OPT_FLAG_FORCE},
13:54 aliguori joined #gluster
13:54 JoeJulian So I grep for that and find: #define VKEY_MARKER_XTIME         GEOREP".indexing"
13:55 JoeJulian So then I have to grep another #define...
13:55 JoeJulian :P
13:56 zoldar "geo-replication.indexing cannot be disabled while geo-replication sessions exist" - and geo-repl status is empty (geo-repl master is down). I'm puzzled...
13:58 zoldar I suppose that removing /var/lib/glusterd/geo-replication/{volname}-volume is not the best idea?
14:00 JoeJulian What's the exact stop error?
14:01 zoldar when trying to stop geo-replication?
14:01 plarsen joined #gluster
14:03 zoldar geo-replication session between mail-volume & ssh://geoaccount@backupib::mail-slavevol not active
14:03 zoldar geo-replication command failed
14:03 vipkilla is there a way to force the client to write a certain user?
14:03 vipkilla like NFS
14:04 JoeJulian vipkilla: no
14:04 vipkilla k thanks
14:04 stopbit joined #gluster
14:04 Machske_ joined #gluster
14:05 Machske_ hi all, got a question which puzzles me a bit
14:05 JoeJulian zoldar: hmm, there's no force option for that either. Darn. You may be right about deleting the geo-replication/{volname}-volume
14:05 Machske_ I've got a 2 node gluster setup in replicate mode with about 10 virtual machine disc images
14:06 deepakcs joined #gluster
14:06 Machske_ when a node reboots, self heal is triggered for the changed disc images, so far so good, keep everything in sync
14:07 Machske_ but during the self heal, most of the virtual machines are stuck waiting on IO. When self heal is finished, this is solved
14:07 JoeJulian 3.2?
14:07 Machske_ 3.3
14:07 JoeJulian well that shouldn't be.
14:07 Machske_ indeed I know this is "normal" in 3.2, but I thought this was modified in 3.3
14:09 Machske_ do I have to set some special volume options or so ?
14:09 JoeJulian Have you looked at server and/or network loads when this happens? Could you be hitting hardware limits?
14:09 JoeJulian Are your servers also clients?
14:10 Machske_ Network load and disc load seems fine, busy but fine.
14:10 Machske_ The servers are indeed also clients
14:10 Machske_ they mount the gluster volume from localhost
14:10 JoeJulian Try using the deadline scheduler.
14:11 JoeJulian which reminds me... I need to change that on my new server...
14:12 Machske_ ok thx for the tip!
14:12 * johnmark files notes re: write up blog post on different schedulers
14:13 JoeJulian johnmark: still on for the meetup today?
14:13 johnmark JoeJulian: yup. about to send a note ot the list
14:13 JoeJulian yay. finally get to play with my new toy.
14:13 Machske_ is it normal that the selfheal process will read the complete contents of all files marked for healing ? We see a lot of reading during this process.
14:15 wushudoin joined #gluster
14:15 JoeJulian it's doing a differential heal. Some people like to set cluster.data-self-heal-algorithm full
14:15 Machske_ what would be the difference ?
14:16 Machske_ does full mean: just copy it over ? :)
14:16 JoeJulian Yep
14:16 Machske_ reading the man page ... :)
14:17 Machske_ is the reset algoritme not more interesting ?
14:17 zoldar JoeJulian: it seems that /var/lib/glusterd/geo-replication is not the only place where session information is kept. After starting gluster, the same geo-replication sessions were brought up despite the directory being empty (besides keys for ssh)
14:19 overclk joined #gluster
14:20 JoeJulian see if it'll take the stop now?
14:21 zoldar JoeJulian: already tried, nop
14:21 zoldar e
14:21 kkeithley joined #gluster
14:22 JoeJulian There's probably something also in /var/lib/glusterd/vols/$vol/info
14:22 JoeJulian And, of course, glusterd should be stopped when you mess with those.
14:23 JoeJulian on both servers. And the changes need to be synched between servers too.
14:23 zoldar hmm, ok
14:24 JoeJulian gotta run. be back in a few hours.
14:26 zoldar thanks for all the feedback so far
14:30 jbrooks joined #gluster
14:36 kkeithley anyone look at the _new_ Debian Squeeze .debs and confirm that they're okay? (wiqd?)
14:36 ekuric1 joined #gluster
14:46 glusterbot New news from newglusterbugs: [Bug 830134] NFS Mount doesn't report "I/0 Error" when a file is in split-brain state <https://bugzilla.redhat.com/show_bug.cgi?id=830134>
14:49 plarsen joined #gluster
14:55 jdarcy Inside joke: the starting "generation" for inodes in that patch is 1965 (my birth year).
14:56 kkeithley dang, you're old
14:56 jdarcy Almost as old as dirt.  ;)
15:05 wiqd kkeithley: still seem to be getting the libssl1.0.0 dependency
15:05 wiqd looks like it's coming from http://download.gluster.org/pub/gluster​/glusterfs/LATEST/Debian/squeeze.repo/d​ists/squeeze/main/binary-amd64/Packages
15:05 kkeithley well sugar
15:06 13WABN59V left #gluster
15:06 gbrand_ joined #gluster
15:06 kkeithley thanks for checking
15:07 wiqd you're welcome, just shout if/when you want me to test, got a vm here with your name on it :)
15:07 kkeithley :-)
15:11 kkeithley hmmm
15:11 gbrand_ joined #gluster
15:12 cjbehm joined #gluster
15:13 kkeithley can't fire up my vm just now. I'm copying all my vm images from the old 2-drive-striped LVM volume to a brand new 4TB SAS drive. Gonna be a little while
15:16 cjbehm Hello - we have a server that we want to replace. It's online, but using replace-brick hammers the CPU on the server being replaed and hurts availability. I believe we can copy the export to a new server (copying xattr and not .glusterfs) and configure the new server with the same hostname as the server being replaced. Does anyone know if that will or won't work?
15:16 faizan joined #gluster
15:16 ekuric joined #gluster
15:17 ekuric joined #gluster
15:17 gbrand_ joined #gluster
15:17 cjbehm I was in earlier and JoeJulian offered some good advice on replace-brick and that we should be able to pre-load data if we had to, but this is a lengthy enough process that I figured I'd see what experts had to say before we went about getting some downtime to make the final change
15:18 cjbehm or rather, see what experts had to say about our specific method of trying this :)
15:18 MTecknology left #gluster
15:26 daMaestro joined #gluster
15:29 foo_ joined #gluster
15:31 semiosis @later tell lng i'm not always in #gluster, but when I am, I read the scrollback (so just leave me a message :)
15:31 glusterbot semiosis: The operation succeeded.
15:39 faizan joined #gluster
15:39 neofob joined #gluster
15:49 Teknix joined #gluster
15:54 sashko joined #gluster
16:03 Machske_ joined #gluster
16:07 hagarth joined #gluster
16:25 H__ yay, rebalance part1 is ready after 7 days. it ran to 1649641
16:28 chouchins joined #gluster
16:36 jbrooks joined #gluster
16:43 lng joined #gluster
16:44 lng Hi! How can I reconfigure Gluster to use hostnames instead of IPs?
16:46 lng I have already probbet it with hostnames, but IPs are still remain in /var/lib/glusterd
16:46 lng any document on that?
16:47 Bonaparte left #gluster
16:53 bala joined #gluster
16:53 vimal joined #gluster
16:54 semiosis ~hostnames | lng
16:54 glusterbot lng: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
16:54 semiosis if you already have volumes defined then it's complicated
16:55 semiosis i've never done that, but i guess you could use replace-brick, possibly, or stop everything and go in & edit volfiles by hand
16:55 semiosis see ,,(replace) first link for ways to use replace-brick
16:55 glusterbot Useful links for replacing a failed server... if replacement server has different hostname: http://community.gluster.org/q/a-replica-no​de-has-failed-completely-and-must-be-replac​ed-with-new-empty-hardware-how-do-i-add-the​-new-hardware-and-bricks-back-into-the-repl​ica-pair-and-begin-the-healing-process/ ... or if replacement server has same hostname:
16:55 glusterbot http://www.gluster.org/community/docum​entation/index.php/Gluster_3.2:_Brick_​Restoration_-_Replace_Crashed_Server
16:59 faizan joined #gluster
17:17 faizan joined #gluster
17:21 bjoernt joined #gluster
17:23 Technicool joined #gluster
17:42 faizan joined #gluster
17:48 Mo___ joined #gluster
17:51 Fabiom joined #gluster
17:52 ctria joined #gluster
17:55 jrossi joined #gluster
17:58 jrossi should read options complete hold up write operations on a two node replica cluster?  I am rsyncing some files to gluster (long term storage) and if I do an ls with in the gluster partation the rsync stops and will begin again when ls has completed listing files.   This all happens very slowly.
17:59 kkeithley joined #gluster
18:00 Technicool joined #gluster
18:00 Mo___ Is there a upper limit on how many mounts we can have with gluster?
18:20 overclk joined #gluster
18:25 plarsen joined #gluster
18:26 y4m4 joined #gluster
18:33 Pushnell_ joined #gluster
18:35 H__ jrossi: that's odd. I've never seen that. (just a gluster user here)
18:42 Fabiom is Gluster Storage Platform only available as a commercial offering and now RedHat Storage ? The version available to download is GlusterFS only, thus you need to install your distro and add GlusterFS packages
18:43 Technicool Fabiom, GSP is deprecated
18:43 Technicool are you looking for a GUI?
18:45 Fabiom GUI would be nice. But turnkey distro is more interesting to me.
18:49 Fabiom in early stages of research. But I understand that GSP was like Openfiler or Nexenta. Distro and App in one
18:50 elyograg Fabiom: I've found that it's very easy to do a simple fedora or centos (rhel if you want paid support), add a repo, and install gluster.
18:50 Technicool fabiom, FC17 includes Gluster 3.2.7 by default
18:51 Technicool GSP was really just meant as a GUI for users who weren't comfortable with the command line
18:51 Technicool if a GUI is required, it is possible to manage gluster via the ovirt GUI
18:51 Fabiom ok agreed. I have installed in Centos6.3 was easy enough.
18:52 Fabiom thanks I will take a look at ovirt GUI
18:54 LW__ left #gluster
18:56 kkeithley Does the oVirt GUI have gluster support? I didn't think it did yet.
18:57 VisionNL jdarcy: I was gone yesterday at the time, yes, we;ll try to do this. Including the gdb trace if posible. We'll need to set it up on a high performance VM node to probably get this.
18:58 manik joined #gluster
19:03 kkeithley johnmark: hah, my camera works, just not in google hangout. And when you wanted me to talk about 3.3.1 I couldn't get the mike to turn on, although I turned it off and on several times earlier.
19:03 kkeithley google software ftw. Not!
19:03 kkeithley Which is funny because video worked in google chat last time I had a reason to use it.
19:06 wN joined #gluster
19:07 kkeithley wikd (and others): now that I've migrated my vm images to new storage——  There's a newer squeeze.repo on download.gluster.org now. I hope that now — finally — there are no references to libssl1.0.0.
19:08 Technicool joined #gluster
19:08 kkeithley The Packages file you mentioned earlier now does not have 1.0.0.
19:16 vipkilla left #gluster
19:17 manik joined #gluster
19:23 wN_ joined #gluster
19:35 semiosis @ovirt
19:35 semiosis @gluster storage platform
19:35 glusterbot semiosis: I do not know about 'gluster storage platform', but I do know about these similar topics: 'What is the difference between Gluster Storage Platform (GSP) and GlusterFS?'
19:35 semiosis @gsp
19:35 glusterbot semiosis: I do not know about 'gsp', but I do know about these similar topics: 'What is the difference between Gluster Storage Platform (GSP) and GlusterFS?'
19:36 semiosis @What is the difference between Gluster Storage Platform (GSP) and GlusterFS?
19:36 glusterbot semiosis: GlusterFS and Gluster Storage Platform (SP) are not the same thing. SP bundles GlusterFS with Linux and a pretty GUI as a bootable image. No command line access is available. SP, though open source, is designed as a closed system. GlusterFS is the clustered file sharing system that SP uses, and is available for many unix type operating syste
19:36 semiosis glusterbot: meh
19:36 glusterbot semiosis: I'm not happy about it either
19:36 semiosis @gui
19:36 glusterbot semiosis: I do not know about 'gui', but I do know about these similar topics: 'git', 'gfid', 'gmc'
19:36 semiosis there it is
19:36 semiosis @gmc
19:36 glusterbot semiosis: The Gluster Management Console (GMC) has been discontinued. If you need a pretty gui to manage storage, support for GlusterFS is in oVirt.
19:36 pdurbin semiosis: i sit in awe of glusterbot
19:36 semiosis wait a sec... i thought there was a factoid with a link to the ovirt stuff
19:38 semiosis learn oVirt as http://wiki.ovirt.org/wiki​/Features/Gluster_Support
19:38 glusterbot Title: Features/Gluster Support - oVirtWiki (at wiki.ovirt.org)
19:38 semiosis @ovirt
19:38 semiosis @oVirt
19:38 semiosis @learn oVirt as http://wiki.ovirt.org/wiki​/Features/Gluster_Support
19:38 glusterbot semiosis: The operation succeeded.
19:38 semiosis @ovirt
19:38 glusterbot semiosis: http://wiki.ovirt.org/wiki​/Features/Gluster_Support
19:38 semiosis @ovirt
19:38 glusterbot semiosis: http://wiki.ovirt.org/wiki​/Features/Gluster_Support
19:38 semiosis ok there we go
19:39 t35t0r joined #gluster
19:39 t35t0r joined #gluster
19:39 stickyboy joined #gluster
19:52 ctria joined #gluster
19:52 Technicool joined #gluster
19:53 ctria joined #gluster
20:00 aliguori joined #gluster
20:07 sashko joined #gluster
20:08 Alpinist joined #gluster
20:34 ctria joined #gluster
20:39 dshea joined #gluster
20:43 ctria joined #gluster
20:43 ctria joined #gluster
20:44 jdarcy "One does not simply fill binders with women"  LOLOLOL
20:53 t35t0r http://www.quickmeme.com/meme/3rf5to/
20:53 glusterbot Title: Boromir - one does not simply fill binders with women (at www.quickmeme.com)
20:55 stickyboy Boromir meme... so hilarious.
21:00 wiqd kkeithley: new problem, there is now a dependency on libssl0.9.8 (>= 0.9.8m-1) which is good, and libssl (>= 0.9.8) which doesn't exist
21:00 wiqd Depends: libc6 (>= 2.8), libibverbs1 (>= 1.1.2), libssl0.9.8 (>= 0.9.8m-1), libssl (>= 0.9.8)
21:10 hattenator joined #gluster
21:18 chandank|work joined #gluster
21:20 chandank|work JoeJulian, Are you aware of some bug in gluster with ext4 file system?
21:20 tryggvil joined #gluster
21:21 hattenator It's with the distribute plugin
21:21 hattenator @ext4
21:21 glusterbot hattenator: Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
21:21 hattenator plugin is the wrong word, but meh
21:22 chandank|work so will it impact if I am not using NFS?
21:23 hattenator I think so
21:23 hattenator but it will not impact you if you aren't using the distribute thing across multiple bricks
21:23 hattenator the pseudo-raid-0
21:23 hattenator replication works fine
21:25 hattenator I'm still using ext4 without real issue for my little project.  RedHat wants a few thousand dollars per year for XFS support.  I can bug my boss for that or just compile xfs-progs myself unsupported, but for now ext4 works fine since I'm not using distribute
21:26 hattenator If you're using a distribution that didn't strip XFS out, you probably should use that, though.
21:26 hattenator XFS does take more thought and configuring to set up than EXT4, though.  I don't have experience with that.
21:53 chandank|work thanks. I am planning to use replication
21:54 chandank|work so I believe ext4 should be safe to use.
21:59 Technicool joined #gluster
22:08 H__ is the load penalty of rebalance-step2 the same as step1 ? (first results seem to hint that)
22:09 Machske_ joined #gluster
22:19 t35t0r centos plus kernel + jfs + gluster
22:37 plarsen joined #gluster
22:38 quillo joined #gluster
23:07 Fabiom xfs is in RHEL/Centos6 - modprobe xfs and yum install xfsprogs xfsdump
23:10 hattenator It's in Centos6.  It's explicitly removed from RHEL6.
23:11 hattenator you can grab xfsprogs from centos and probably install it on RHEL6, or compile it yourself, but it isn't available from yum.
23:12 hattenator You need to buy the "Scalable Storage Channel" for RHEL if you want it from yum and with support.
23:13 ackjewt joined #gluster
23:14 Fabiom thanks for the clarification
23:14 hattenator Rumor is that it'll be included in the base for RHEL7
23:15 avati joined #gluster
23:41 Fabiom need some help understanding. Distributed Replicated Volumes 1brick per server, with replica count "2", 1 volume. Is it like RAID10 for redundancy. server1/server2 replicaset-1 server3/server4 replicaset-2 etc.. So if I lose server1 and server2 data lost. But I could lose server1 and server3 and still be ok ?
23:56 sashko joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary