Camelia, the Perl 6 bug

IRC log for #gluster, 2012-10-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:01 JoeJulian JFK: I think there was some decaffeinated moment in your own progression that led to that. Perhaps the brick wasn't mounted when the volume was recreated. Then when it was mounted later, it had the wrong volume id. If resyncing the volume data doesn't cure it, then that's going to be my guess.
00:08 JFK but it was working quite ok for some time (2-3 days) and everything was fine
00:09 nueces joined #gluster
00:09 JoeJulian JFK, then the server was rebooted and the brick (that was accidentally left unmounted when you were planning on just reformatting it) mounted from fstab and voila.
00:09 JoeJulian At least that's one possible scenario...
00:10 JoeJulian As long as I'm creating fiction, there was probably an NFL cheerleader that was distracting you at the same time.
00:10 JFK ;-)
00:11 elyograg I figured out how to get the proper URL into proxy-server.conf so that remotely accessing the swift volume from a machine not running proxy-server works right.  now I can use the swift command from remote servers to access the object store.  but cyberduck still doesn't work.  Did a packet capture and stuck my ssl key into it.  cyberduck is GETting the /v1.0 url, but apparently /auth/v1.0 is what's required.  HTTP/1.1 412 Precondition Failed
00:11 JFK cheerleader maybe not but this could be my wife :-)
00:11 tc00per JoeJulian: Did you mean to say... "On your servers is /srv/glusterfs/bricks/gvol.0*/.glusterfs/0​0/00/00000000-0000-0000-0000-000000000001 as symlink to ../../.. on all of them?"
00:12 JoeJulian Ah, glad I pointed you there. Yes, that was what I was actually thinking.
00:12 tc00per Then... Yup... symlinks on all servers.
00:13 JFK ok, to sum up: if I want it up and running again without too much effort i should create new volume and rsync data between servers, shouldnt i?
00:14 elyograg wow, it got late.
00:14 JoeJulian JFK: You have access to all the correct data with those bricks missing?
00:14 JoeJulian elyograg: It has a tendency to do that... :D
00:15 JoeJulian elyograg: Some applications, not sure about cyberduck, have a way of setting the auth url.
00:15 JFK yes, even on those machines with offline brics mounts are ok
00:16 tc00per JoeJulian: subdir replicas has drop by one from 3->2 or 2->1 on unmount/mount of glusterfs client.
00:16 JoeJulian Ok, if your data's intact, I would probably just format those offline bricks and let replicate fix them.
00:16 JFK but everything is taken from this online brick via network
00:16 tc00per s/has/have
00:16 tc00per s/has/have/
00:16 JFK you mean delete whole partition?
00:16 glusterbot What tc00per meant to say was: JoeJulian: subdir replicas have drop by one from 3->2 or 2->1 on unmount/mount of glusterfs client.
00:17 elyograg JoeJulian: It does have a "path" setting, but I don't think that affects the initial authentication URL.  Perhaps if someone can just point me at docs to tell me how to enable s3 access in gluster-swift, I won't have to worry about this at all.
00:17 JoeJulian tc00per: I think that's a self-heal bug that's fixed in master. I expect it'll be in 3.0.1
00:17 JoeJulian JFK: I'm thinking "mkfs.xfs -i size=512"
00:18 tc00per JoeJulian: do you mean 3.3.1?
00:18 JoeJulian yes
00:18 JoeJulian I think I've reached the limits of my brain's cores. multithreading's failing.
00:18 tc00per :)
00:18 JFK JoeJulian: sounds radical :-)
00:19 JFK well i do have backup... somewhere.. i hope ;-)
00:19 JoeJulian I'm also writing a blog post, debugging an error in my blog, modifying a python script to use the new rackspace api v2.0...
00:19 tc00per JFK: doesn't glusterfs replicate do backup for you? ;)
00:20 tc00per JoeJulian: As I said earlier... you are a glusterfs god. :)
00:20 JoeJulian hehe
00:20 JFK tc00per: i prefer rather traditional methods - standalone machine in other datacenter and rsync  :-)
00:21 JoeJulian Which is great unless you rsync bad data to it.
00:21 * JoeJulian had to deal with that today.
00:21 elyograg headed home.  I'll be idling here.
00:22 tc00per All while helping the glusterfs noobz... :)
00:23 JFK JoeJulian: answer is backup-dir incremental backups
00:23 JFK you have at least week to notice error
00:26 JoeJulian We do a double-backup thing... We backup at 23:00 to get all the big stuff synced up if it's not done by 4am, we kill it and then do another rsync at 4:00. This actually works pretty good to get everything done on time without having to pick and choose which things need to get backed up when.
00:27 JoeJulian The problem is, there's one machine that reboots at 3:00. When it did, it mounted the previously failed mirror (need to fix that notification!!!) and apparently failed the good drive. This gave me stale data during that second rsync.
00:28 JoeJulian So my only really good backup was a day old.
00:28 JoeJulian I'm thinking about trying out btrfs for my backup drives...
00:31 blendedbychris joined #gluster
00:31 blendedbychris joined #gluster
00:33 JFK if i'm not mistaken backup-dir should move first rsync  data to another folder during second rsync in such case
00:34 JoeJulian The backup disks (external) don't have enough space for that.
00:35 JoeJulian Maybe that should be the change... we'll be discussing this tomorrow I'm sure.
00:39 JFK for me this is optimal because we do have a lot of data but most of files do not change for long period of time so most of rsyncs just moves deleted/changed files to date-named directory and adding new ones to 'current'
00:40 JFK those date-named are only few megabytes size so there is no problem at all
00:46 JoeJulian http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/
00:46 glusterbot Title: How to expand GlusterFS replicated clusters by one server (at joejulian.name)
00:50 JoeJulian stigchristian, tc00per ^^
00:51 JFK ok, i think i should go to bed before i make another mistake. it is 3am here :-)
00:51 JoeJulian Sleep well
00:51 JFK JoeJulian: really do thaks for your help.
00:51 JoeJulian You're welcome. :)
00:51 JFK tc00per is right you are a glusterfs god :)
00:53 ankit9 joined #gluster
00:53 JFK bye
00:53 JFK left #gluster
01:02 tc00per JoeJulian: Thanks for the blog post.
01:18 rwheeler joined #gluster
01:19 lkthomas joined #gluster
01:19 kore joined #gluster
01:20 zwu joined #gluster
01:23 kore joined #gluster
01:41 kevein joined #gluster
01:49 ankit9 joined #gluster
01:57 zwu joined #gluster
02:06 zwu joined #gluster
02:17 layer3 joined #gluster
02:20 harish_ joined #gluster
02:30 dec joined #gluster
02:34 sunus joined #gluster
02:50 sgowda joined #gluster
02:54 bulde1 joined #gluster
03:20 sensei joined #gluster
03:45 atrius so... gluster for VM backing storage... still a bad idea? :D
03:50 kshlm joined #gluster
03:51 m0zes atrius: in my experiece, it depends on the i/o load in the vm. I run low i/o vms on glusterfs 3.2 (iirc 3.3 does better with granular locking in a replica environment, but not too much better for i/o in a vm)
03:52 atrius m0zes: i presume the main concern is that under high I/O load situations gluster can't keep up with keeping things synced up?
03:53 m0zes that and going through user->kernel->user->kernel cycles to write/read to a vm disk slows things down a bit ;)
03:53 atrius there is that
03:54 atrius i wonder if it would just be better to use NFS for this instead and skip gluster all together
03:54 rferris joined #gluster
03:54 m0zes for me, I use small vm disks on a dist+replicate volume, and any serious i/o load gets pushed to a gluster mount within the vm. that has worked fairly well for me.
03:55 atrius that would skip a bit of redirection after all
03:56 m0zes if you push all i/o to a vm disk, you get hosts pissed off like so: http://ganglia.beocat.cis.ks​u.edu/?c=Beocat&h=tyche
03:56 glusterbot Title: tyche Host Report (at ganglia.beocat.cis.ksu.edu)
03:57 m0zes that is my ganglia vm. it isn't doing much except reading/writing to rrd files.
03:57 atrius i see
03:57 m0zes constant moderate load, fairly poor write performance, and almost reasonable read performance.
03:58 atrius hmmm.... so it works better having the VMs mount things directly then?
03:59 m0zes from my experience, yes. I haven't had time to migrate /this/ vm to a volume mounted withing the guest, but with others it has worked well.
03:59 atrius how does that impact VM migration between hosts?
04:02 m0zes vm migration seems to work just fine. kvm doesn't like to migrate vms with disk storage that isn't mounted with o_direct (unless you use --force), so you'll want a 3.4 kernel or newer (or @liboindirect). FUSE before kernel 3.4 doesn't support o_direct.
04:02 m0zes glusterbot: @odirect
04:02 atrius lol... so you're telling me that migration using a fuse system doesn't work worth a damn prior to a 3.4 kernel?
04:03 m0zes https://github.com/avati/liboindirect
04:03 glusterbot Title: avati/liboindirect · GitHub (at github.com)
04:04 m0zes it works, and I have tested it on numerous occasions. kvm gets won't do it without --force or liboidirect
04:04 atrius ah, okay
04:05 atrius well.. this is fun.. i think my first attempt just failed... my vm images are acting like they're corrupted
04:05 atrius thankfully they're scratch instances
04:07 m0zes hooray. like I said, in my experience it has worked. YMMV...
04:07 rferris joined #gluster
04:08 * m0zes runs Gentoo, so doesn't experience certain oddities others face. I certainly face a good number of my own oddities, though.
04:08 atrius yeah, i remember the occasional "it works on gentoo!" and the "it fails only on gentoo! :-(" moments :D
04:09 rferris joined #gluster
04:09 m0zes Gentoo works very well for my use case. I wouldn't recommend it for most, though.
04:10 * m0zes runs a heterogenous hpc cluster. it is nice to have images tuned for specific hardware classes.
04:10 atrius yeah, i rarely had problems with it back in the day
04:21 sripathi joined #gluster
04:24 jays joined #gluster
04:29 vpshastry joined #gluster
04:37 shylesh joined #gluster
04:41 mdarade joined #gluster
04:44 raghu joined #gluster
04:45 ramkrsna joined #gluster
04:45 ramkrsna joined #gluster
04:48 Humble joined #gluster
04:53 zwu joined #gluster
05:05 atrius ick... having the system more or less hang doing an 'ls' on both clients is no fun
05:09 Humble joined #gluster
05:18 hagarth joined #gluster
05:20 kshlm joined #gluster
05:20 kshlm joined #gluster
05:23 faizan joined #gluster
05:24 shylesh joined #gluster
05:32 JoeJulian atrius, m0zes: http://www.redhat.com/archives/lib​vir-list/2012-August/msg01625.html
05:32 glusterbot Title: [libvirt] [RFC PATCH v1 0/2] Qemu/Gluster support in Libvirt (at www.redhat.com)
05:37 samppah awww.. when it is going to be available
05:40 JoeJulian I have no idea, but they've demo'd a working system.
05:42 samppah yep, it looks promising indeed
05:42 raghu joined #gluster
05:42 samppah can't wait that there is a support for it in ovirt also :)
05:45 harish joined #gluster
05:45 sgowda joined #gluster
05:45 shylesh joined #gluster
05:45 wushudoin joined #gluster
05:46 mdarade left #gluster
05:47 sunus joined #gluster
05:48 sunus joined #gluster
05:50 ngoswami joined #gluster
05:50 sunus1 joined #gluster
05:51 Humble joined #gluster
05:53 shylesh joined #gluster
05:58 shylesh_ joined #gluster
06:04 vpshastry1 joined #gluster
06:04 deepakcs joined #gluster
06:04 kaushal_ joined #gluster
06:04 kaushal_ joined #gluster
06:07 bulde1 joined #gluster
06:07 bulde joined #gluster
06:08 wmp_ joined #gluster
06:11 pkoro joined #gluster
06:11 mdarade1 joined #gluster
06:11 Humble joined #gluster
06:12 bulde joined #gluster
06:14 mo joined #gluster
06:14 sgowda joined #gluster
06:22 wmp_ left #gluster
06:24 lkoranda joined #gluster
06:26 sashko joined #gluster
06:45 mdarade joined #gluster
06:52 ngoswami joined #gluster
06:53 jays joined #gluster
06:54 andreask joined #gluster
06:58 bala joined #gluster
07:01 ekuric joined #gluster
07:05 ondergetekende joined #gluster
07:06 92AABM2DI joined #gluster
07:07 cattelan joined #gluster
07:09 ctria joined #gluster
07:09 raghu joined #gluster
07:12 Nr18 joined #gluster
07:15 ankit9 joined #gluster
07:20 stickyboy joined #gluster
07:26 tjikkun_work joined #gluster
07:32 sunus1 joined #gluster
07:34 ctrianta joined #gluster
07:40 Nr18 joined #gluster
07:42 ngoswami_ joined #gluster
07:43 lkoranda joined #gluster
07:43 TheHaven joined #gluster
07:51 bulde joined #gluster
07:56 Humble joined #gluster
07:57 TheHaven joined #gluster
07:57 aberdine joined #gluster
08:02 dobber joined #gluster
08:05 shylesh joined #gluster
08:10 vpshastry joined #gluster
08:13 ngoswami joined #gluster
08:16 gbrand_ joined #gluster
08:18 hagarth joined #gluster
08:27 badone joined #gluster
08:27 ntt joined #gluster
08:27 ntt hi
08:27 glusterbot ntt: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
08:28 kshlm joined #gluster
08:28 kshlm joined #gluster
08:29 ntt i'm using glusterfs 3.3.0 with replica = 2 but it seems that i can access in read-only mode (mounted with native glusterfs client). I cannot write the volume. It's a permission problem?
08:48 rgustafs joined #gluster
08:53 ngoswami_ joined #gluster
08:53 vimal joined #gluster
08:54 vpshastry1 joined #gluster
08:55 faizan joined #gluster
08:56 ramkrsna joined #gluster
09:08 shylesh joined #gluster
09:11 badone_home joined #gluster
09:13 shylesh joined #gluster
09:17 zArmon joined #gluster
09:20 shylesh joined #gluster
09:22 vpshastry joined #gluster
09:25 TheHaven joined #gluster
09:25 badone joined #gluster
09:28 Venkat joined #gluster
09:32 ankit9 joined #gluster
09:36 badone_home joined #gluster
09:43 flowouf ntt ?
09:43 flowouf hello
09:43 glusterbot flowouf: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
09:43 flowouf what do u see when u use gluster volume info ?
09:45 stickyboy glusterbot: iptables
09:45 stickyboy !iptables
09:45 ndevos @ports
09:45 glusterbot ndevos: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
09:45 stickyboy :)
09:46 faizan joined #gluster
09:47 TheHaven joined #gluster
09:48 vikumar joined #gluster
09:48 ngoswami_ joined #gluster
09:49 ramkrsna_ joined #gluster
09:51 Jippi joined #gluster
09:58 tryggvil joined #gluster
09:59 ngoswami joined #gluster
09:59 tryggvil joined #gluster
10:10 oneiroi joined #gluster
10:13 shireesh joined #gluster
10:14 Triade joined #gluster
10:15 oneiroi joined #gluster
10:17 badone_home joined #gluster
10:28 raghu joined #gluster
10:29 faizan joined #gluster
10:37 shylesh joined #gluster
10:39 bulde1 joined #gluster
10:40 ngoswami joined #gluster
10:45 rosco On linux, i use the awesome glusterfs mount option for failover etc.
10:46 rosco How can one achieve this failover feature on windows?
10:46 rosco Do I have to work with virtual ip adresses?
10:47 rosco Or is there some windows network drive director for failover?
10:49 vpshastry1 joined #gluster
10:52 oneiroi joined #gluster
10:59 ramkrsna_ joined #gluster
11:00 nueces joined #gluster
11:01 shylesh_ joined #gluster
11:03 sgowda joined #gluster
11:10 stickyboy rosco: Which mount option is this?  Sounds like something I need...
11:10 stickyboy (not for Windows)
11:12 rosco stickyboy: I used for my setup: mount -t glusterfs fileserver1:/STATIC-DATA /mnt/static-data/
11:13 rosco This will initially connect to 1 fileserver but will be failover to fileserver2 when not reachable
11:13 dobber Hello, I got my InfiniBand cards installed and ready. How do I instruct glusterfs to work over IB?
11:15 rosco dobber: Is this something you need? http://download.gluster.org/pub/gluster/gl​usterfs/3.2/Documentation/IG/html/sect-Ins​tallation_Guide-Installing-Configure.html
11:15 glusterbot Title: 3.4. Configuring GlusterFS to work over InfiniBand (at download.gluster.org)
11:16 dobber i already compiled it with rdma support
11:16 stickyboy rosco: Ah, so it fails over to the other peered servers automatically.
11:16 rosco stickyboy: yes.
11:16 stickyboy rosco: And I just realized, I'm going to be using distribute anyways.
11:16 badone_home joined #gluster
11:16 stickyboy rosco: I guess that's the benefit of the FUSE client, which Windows doesn't have?
11:17 rosco stickyboy: Well, yes. I am searching for a failover solution for windows that works like that.
11:18 shireesh joined #gluster
11:18 stickyboy rosco: Gotcha.  Sorry, I can't help.
11:19 glusterbot New news from newglusterbugs: [Bug 864401] [glusterfs-3.3.1qa3]: glusterfs client asserted <https://bugzilla.redhat.com/show_bug.cgi?id=864401>
11:19 stickyboy I wonder if protocol stacking would help.
11:19 stickyboy ie, mount FUSE and then export the resulting mount via NFS.
11:19 stickyboy People have been doing interesting things with that lately...
11:21 Heebie joined #gluster
11:23 kkeithley1 joined #gluster
11:24 rosco dobber: I guess you need this: gluster volume create NEW-VOLNAME [replica COUNT] [transport tcp | rdma | tcp,rdma] NEW-BRICK:/EXPORT-PATH NEW-BRICK:/EXPORT-PATH...
11:24 rosco dobber: And the transport would be rdma
11:24 dobber i think you are probably right
11:24 dobber now i have to find how to change it on a working volume
11:24 dobber thanks
11:29 rosco dobber: Maybe you can create the volume on the same EXPORT-PATHS again, but with a different transport type.
11:29 dobber of find where is the configuration file ?
11:30 hagarth joined #gluster
11:31 dobber s/of/or/
11:31 glusterbot What dobber meant to say was: or find where is the configuration file ?
11:33 rosco dobber: I don't know if you can change the transport type. A save bet would be to make a new connection and then delete the old connection (or the other way around.)
11:33 dobber i don't feel comfortbale deleting my volume
11:34 dobber i guess i have to detach the peers too :(
11:34 dobber and re-atach them with the ib network
11:40 dobber ok maybe i have to test it with a test volume first anyway
11:46 sashko joined #gluster
11:53 shylesh_ joined #gluster
11:54 tryggvil joined #gluster
11:56 Venkat joined #gluster
11:56 Nr18 joined #gluster
11:57 ondergetekende joined #gluster
12:02 dobber Is there a way to specify witch Infiniband port to use ?
12:06 FU5T joined #gluster
12:08 dobber rpc-transport/rdma: by default use infiniband port 1
12:08 dobber how do i change this
12:09 vpshastry joined #gluster
12:14 balunasj joined #gluster
12:15 flowouffff joined #gluster
12:15 dobber 2012-10-09 11:47:04.264572] W [rdma.c:3766:gf_rdma_get_device] 0-rpc-transport/rdma: On device mthca0: provided
12:15 dobber port:1 is found to be offline, continuing to use the same port
12:15 dobber there should be an active port 2
12:30 Venkat joined #gluster
12:32 vpshastry left #gluster
12:46 andreask joined #gluster
12:48 hagarth joined #gluster
12:49 Guest42583 left #gluster
12:50 pdurbin joined #gluster
12:56 jays joined #gluster
12:57 tryggvil_ joined #gluster
12:58 ramkrsna joined #gluster
12:58 ramkrsna joined #gluster
12:58 dbruhn__ joined #gluster
13:03 shireesh joined #gluster
13:04 bulde joined #gluster
13:09 sashko joined #gluster
13:15 HavenMonkey joined #gluster
13:19 glusterbot New news from newglusterbugs: [Bug 864499] gluster volume geo-replication start --xml outputs wrong xml structure <https://bugzilla.redhat.com/show_bug.cgi?id=864499> || [Bug 864502] gluster volume geo-replication stop --xml outputs wrong xml structure <https://bugzilla.redhat.com/show_bug.cgi?id=864502>
13:21 vpshastry joined #gluster
13:22 ankit9 joined #gluster
13:22 jays joined #gluster
13:27 rwheeler joined #gluster
13:32 wN joined #gluster
13:32 Nr18 joined #gluster
13:33 vpshastry left #gluster
13:37 djs joined #gluster
13:40 tryggvil joined #gluster
13:44 plarsen joined #gluster
13:49 TheHaven joined #gluster
13:49 glusterbot New news from newglusterbugs: [Bug 864503] gluster volume geo-replication config --xml outputs wrong xml structure <https://bugzilla.redhat.com/show_bug.cgi?id=864503> || [Bug 864506] gluster volume geo-replication status --xml outputs wrong xml structure <https://bugzilla.redhat.com/show_bug.cgi?id=864506> || [Bug 864509] gluster volume geo-replication config key value --xml outputs wrong xml structure <https://bugzilla.r
13:51 bulde1 joined #gluster
13:53 bulde1 joined #gluster
13:54 bulde joined #gluster
13:58 bulde joined #gluster
13:58 * jdarcy o_O
14:00 stopbit joined #gluster
14:02 Nr18 joined #gluster
14:05 jbrooks joined #gluster
14:11 noob2 joined #gluster
14:11 deepakcs joined #gluster
14:21 faizan joined #gluster
14:28 lh joined #gluster
14:30 sashko joined #gluster
14:33 kaisersoce joined #gluster
14:33 mdarade left #gluster
14:35 wushudoin joined #gluster
14:49 glusterbot New news from resolvedglusterbugs: [Bug 823154] Brick process crashed while rebalancing distributed-replicate volume with kernel untaring on the mount <https://bugzilla.redhat.com/show_bug.cgi?id=823154>
14:53 Heebie What is kernel untar?  untarring the source code for a Linux kernel?
14:54 pdurbin i would assume so
14:54 johnmark Heebie: that is my assumption
14:54 pdurbin johnmark: jinx
14:54 johnmark teehee
14:54 johnmark :P
15:00 Nr18 joined #gluster
15:01 mweichert joined #gluster
15:01 camel1cz joined #gluster
15:02 camel1cz Howdy guys :-)
15:02 daMaestro joined #gluster
15:02 dobber Hello I think there is a bug in gluster's rdma driver
15:03 dobber how can I submit a bug report?
15:03 JoeJulian You can file a bug report at the following link
15:03 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
15:04 faizan joined #gluster
15:04 mrquesty joined #gluster
15:04 JoeJulian like someone did a couple hours ago at least 5 times! What the...
15:05 sripathi joined #gluster
15:06 JoeJulian rosco: You can use nfs with a floating ip for windows. It'll still break connection when it fails over, but it's as close as windows can get.
15:06 dbruhn joined #gluster
15:06 * camel1cz doesnÂ't like the word bug in the first sentene he read :-D
15:06 mrquesty Hi Guys, does anybody have a clue regarding this topic http://www.mail-archive.com/gluste​r-users@gluster.org/msg10012.html?
15:06 glusterbot Title: [Gluster-users] Setting up Gluster on fully packed hdd's (at www.mail-archive.com)
15:08 JoeJulian rosco: Well... one other somewhat radical idea would be to run a vm with linux inside windows. On that vm, run glusterd and add that machine to the peer group. That will make the vm an nfs server and windows could mount from that. That would give windows a stable tcp connection and the linux vm would have all the advantages of the native client.
15:08 mrquesty From my perspective, it shouldn't be too complicated, but to set some xattr fields and to run heal or similar, but i'm stuck with google and no solutions...
15:09 mrquesty Any ideas?
15:09 dobber I see there is a fix for my issue, but I'm not sure it work...
15:09 dobber https://bugzilla.redhat.com/show_bug.cgi?id=762525
15:09 glusterbot Bug 762525: low, low, ---, fharshav, CLOSED CURRENTRELEASE, Infiniband port handling should be dynamic by picking up from active ports
15:09 dobber it sais that I can specify an active IB port, but does not say how do I do this
15:11 JoeJulian mrquesty: Probably will work if there's room for the creation of the .glusterfs tree and self-healing. It seems to take up some extra space but I haven't determined how much or why.
15:11 semiosis symlinks?
15:11 JoeJulian Nah, those should only be inodes.
15:12 semiosis you're thinking hard links
15:12 JoeJulian symlinks too. They're just dirents.
15:12 semiosis symlinks are actual data, text files which contain the path
15:12 semiosis hm
15:13 JoeJulian No.... <said with incredulity as if it's inconceivable that that much space would be wasted for that>
15:13 JoeJulian jdarcy: ^^ ?
15:13 semiosis when i create a symlink with 'ln -s 123456' ls reports it is 6 bytes
15:14 * jdarcy wakes up slightly.
15:14 adechiaro joined #gluster
15:14 semiosis i'm 99% sure theyre text files with type l
15:14 camel1cz Does anyone glfs on two nodes (replication) in production?
15:14 JoeJulian Well I wouldn't have done it that way.... :)
15:14 jdarcy For files we use hard links in .glusterfs.  Can't do that for directories, so we use symlinks instead.
15:14 camel1cz semiosis: You are right
15:14 JoeJulian Damn. I hate when that happens.
15:14 semiosis :P
15:15 rosco JoeJulian: The windows servers are already a VM :)
15:15 rosco JoeJulian: Inception would not work in this case ;)
15:16 JoeJulian rosco: Oh! Well then if your vm host runs glusterd and is part of the peer group, it'll share nfs and your windows vm's can connect to that.
15:16 semiosis camel1cz: yeah lotsa people use glusterfs replicating between two nodes... whats your question?
15:16 seanh-ansca joined #gluster
15:17 ramkrsna joined #gluster
15:17 JoeJulian Still, the .glusterfs tree used up a big chunk of space during self-heal. I think it had something to do with creating a gfid file before replacing the named file with a hardlink to it, but that's just a guess.
15:18 JoeJulian <mrquesty> Hi Joe, regarding your reply about gluster on a fully packed hdd -> i can't make gluster see files on these two hdd's - e.g. create a gluster replicated volume with 2 peers with these directories, but when i mount the volume - it has 0 (zero) files... Any ideas
15:19 JoeJulian Check the client and brick logs for errors. It normally works.
15:20 camel1cz semiosis: Thxx! ...have spent some time about 2 years ago testing glfs but there was perf problems with a lot of files in directory (maildir)...  I'mjust checking out if there is possibly time to deploy glfs on my mail/web/pgsql servers
15:20 JoeJulian mrquesty: That is considered to be outside the supported practices and gives "unpredictable results". We all do it though.
15:20 mrquesty Joe, i have a simple configuration of 2 peers with 1 partition each, there are just 100 files synced on both. When i run mount command on third node, it doesn't see a single file on mounted drive
15:21 JoeJulian ~pasteinfo | mrquesty
15:21 glusterbot mrquesty: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
15:21 jdarcy If files without GlusterFS xattrs (on all replicas) aren't "imported" when they're seen, I'd consider that a bug.
15:22 mrquesty http://fpaste.org/aNSc/
15:22 glusterbot Title: Viewing Paste #241876 (at fpaste.org)
15:23 JoeJulian mrquesty: Now fpaste "getfattr -m . -d -e hex /DATA" from both please.
15:23 mrquesty Guys, i assume i still should run a set of commands before files appear for Gluster. But not certain, what particular commands i should run ;-)
15:23 JoeJulian No, that's usually enough.
15:23 semiosis iirc when glusterfs encounters two files at the same path and neither has ,,(extended attributes) it does a self-heal using the one with the most recent mtime
15:23 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
15:24 semiosis been a while since i tested that out though, so i may not be remembering correctly, and it may have changed since then too
15:24 semiosis also sets up the missing xattrs on both
15:24 mrquesty http://fpaste.org/4D3Y/
15:24 glusterbot Title: Viewing Paste #241877 (at fpaste.org)
15:25 JoeJulian Well that much worked....
15:26 JoeJulian mrquesty: from the client: unmount, truncate the client log, mount, ls the mountpoint and fpaste the log please.
15:27 camel1cz semiosis: My idea is to create two node replicated glfs and on top of it run openvz driven by heartbeat... and to know if such setup is supported and expected to be production ready.
15:28 * JoeJulian blames a2 for "glfs"
15:29 blendedbychris joined #gluster
15:29 blendedbychris joined #gluster
15:29 mrquesty http://fpaste.org/t7TK/
15:29 glusterbot Title: Viewing Paste #241880 (at fpaste.org)
15:30 JoeJulian iirc, openvz has some issues with fuse filesystems. I guess there are settings to solve that problem but that's about all I know about it.
15:30 mrquesty hmm... i see a files object in a mount point, but it seems to be broken (e.g. red color in ls output)
15:30 JoeJulian mrquesty: E [socket.c:1715:socket_connect_finish] 0-test1-client-1: connection to  failed (Connection refused)
15:30 JoeJulian check your iptables
15:31 mrquesty Joe, it's empty
15:31 mrquesty and both are virtual machines running on my PC
15:31 mrquesty under virtualbox
15:31 semiosis JoeJulian, mrquesty: [2012-10-09 19:29:02.365273] E [afr-common.c:1852:afr_lookup_done] 0-test1-replicate-0: Failing lookup for /files, LOOKUP on a file without gfid is not allowed when some of the children are down
15:32 JoeJulian Right, and that goes back to the "Connection refused"
15:32 semiosis yep
15:32 mrquesty both nodes can see each other properly
15:32 semiosis mrquesty: check iptables *on both servers*
15:33 JoeJulian I notice it doesn't list a port. Make sure glusterd is running on .181
15:33 semiosis that too
15:33 mrquesty http://fpaste.org/OLyk/
15:33 glusterbot Title: Viewing Paste #241883 (at fpaste.org)
15:35 JoeJulian If glusterd is running there, from your client try to telnet 192.168.2.181 24007
15:35 camel1cz JoeJulian: nono, the OpenVZ containers should use the glfs mounted o nOpenVZ hosts... but have no idea if iit's a good setup
15:36 JoeJulian We kind-of have the philosophy that if a setup suits your design requirements then it's a good setup.
15:36 mrquesty hm... i can see files directory in the mount point, but ls command gets stuck
15:36 semiosis mrquesty: what distro/kernel version are your servers running?
15:36 mrquesty semiosis: both are CentOS 6.3 with 2.6.32-279.2.1.el6.centos.plus.x86_64
15:37 semiosis ~ext4 | mrquesty
15:37 glusterbot mrquesty: Read about the ext4 problem at http://joejulian.name/blog/gluste​rfs-bit-by-ext4-structure-change/
15:37 semiosis mrquesty: that's not good :(
15:37 JoeJulian Hmm, good call.
15:37 camel1cz JoeJulian: Nice! Then I'll get this setup up in my lab and will be back soon :-)
15:37 semiosis mrquesty: you'll need to either downgrade to an older kernel or migrate those bricks to XFS to use glusterfs
15:37 mrquesty semiosis: yes, i've read it
15:38 mrquesty semiosis: got it
15:38 camel1cz Thank you, guys... CU later! :-)
15:38 semiosis camel1cz: pleae let us know how it goes
15:38 semiosis please*
15:38 camel1cz semiosis: You bet, I'll :-)
15:38 mrquesty Anyway, thanks to All, i can see all files on mounted point now. Assuming self-heal has helped me :)
15:38 semiosis camel1cz: and it would be really great if you could write a little bit about it too
15:38 Daxxial_ joined #gluster
15:39 mrquesty Thanks again and have a great day!
15:39 semiosis yw
15:39 JoeJulian You're welcome
15:40 JoeJulian I bet glusterd wasn't running on .181.
15:40 borei joined #gluster
15:41 JoeJulian I wish he's confirmed though.
15:41 JoeJulian he'd
15:41 semiosis the whole unable-to-ls-files seemed like a roadblock imo but mrquesty didnt seem bothered by it
15:41 semiosis brb
15:42 ctria joined #gluster
15:42 rodlabs_ joined #gluster
15:44 faizan joined #gluster
15:46 tc00per JoeJulian: Good morning (for me)... back for more. Repaved bricks/gvol. Broken symlink and multiple subdir problems from yesterday are gone. Likely created by me with new peers without DNS or /etc/hosts entries. Lesson learned. Thanks.
15:49 glusterbot New news from newglusterbugs: [Bug 830134] NFS Mount doesn't report "I/0 Error" when a file is in split-brain state <https://bugzilla.redhat.com/show_bug.cgi?id=830134>
15:52 tc00per 4 peers in dist-replx2, only 2 peers have 'active' bricks, just went from 1 x 2 = 2 to 2 x 2 =4 on two peers. Why are all 4 peers being scanned during rebalance?
15:56 tryggvil joined #gluster
16:03 sashko joined #gluster
16:04 tc00per rebalance complete, 9420 (F)iles/ 8515 (S)ymlinks/ 67 (D)irs, 18002 (T)otal. 4530 rebalanced files, 4480 failures. All " E " in rebalance log are "migrate-data failed". All symlinks in 'data' (ie. not .glusterfs) are solid after rebalance. Not sure if failures/" E " are useful information. Ideas?
16:05 dbruhn joined #gluster
16:14 dbruhn joined #gluster
16:20 dbruhn joined #gluster
16:24 sashko joined #gluster
16:32 geggam joined #gluster
16:40 Mo___ joined #gluster
16:44 lh joined #gluster
16:44 lh joined #gluster
16:44 bennyturns joined #gluster
16:45 Humble joined #gluster
16:48 Triade joined #gluster
16:49 aliguori joined #gluster
17:06 rwheeler joined #gluster
17:12 tc00per After taking a 1 x 2 dist-repl glusterfs to 4 x 2, including rebalance per JoeJulians blog post... http://joejulian.name/blog/how-to-expand-gl​usterfs-replicated-clusters-by-one-server/ ... Attempt to remove the data 'tree' resulted in two "cannot remove directory" failures. From glusterfs client all other files/dirs are gone but these two which cannot be removed, even as root. Client log error is "remote operation failed: Directory not empty". Is this spl
17:12 glusterbot Title: How to expand GlusterFS replicated clusters by one server (at joejulian.name)
17:14 cmcdermott1 joined #gluster
17:17 imcsk8 joined #gluster
17:18 Humble joined #gluster
17:18 cmcdermott1 I'm having a weird problem when I try to create a new gluster volume:
17:18 cmcdermott1 [chris@prod01 ~]$ sudo gluster volume create wwwdata replica 2 transport tcp prod01:/gluster/wwwdata prod02:/gluster/wwwdata
17:18 cmcdermott1 Brick: prod01:/gluster/wwwdata, prod02:/gluster/wwwdata one of the bricks contain the other
17:18 cmcdermott1 gluster peer status on both hosts comes back OK - they show each other connected
17:19 cmcdermott1 the only thing I was able to find via the google was a reference to removing extended filesystem attributes, but there are none set in this case - /gluster/wwwdata is a brand new directory with nothing inside (and I've tried several other variations like /gluster-data, all of which generate the same error)
17:29 lh joined #gluster
17:33 hagarth cmcdermott1: are there extended attributes on / by any chance?
17:35 cmcdermott1 no
17:35 cmcdermott1 I did find a solution though
17:35 cmcdermott1 I gave the different directories different names
17:36 atrius so, i was trying to setup gluster between two servers (A and B) with the backing storage being provided by storage server S over iscsi... i created the pool.. mounted the iscsi share on both servers and then remounted the gluster volume somewhere else on both machines... is this a bad way to do this?
17:36 cmcdermott1 so the bricks are prod01:/gluster/brick1 and prod02:/gluster/brick2
17:36 cmcdermott1 that works fine
17:36 cmcdermott1 so I'm thinking maybe it's something new in gluster 3.3 - I think the other hosts where I'm running gluster with bricks that have the same directory names are all gluster 2.x
17:37 semiosis cmcdermott1: you should be able to do it still
17:37 semiosis imho having replicas named the same is a best practice
17:39 davdunc joined #gluster
17:39 davdunc joined #gluster
17:39 elyograg i fixed cyberduck with UFO.  You can manually edit the cyberduck config and change the auth url.  http://trac.cyberduck.ch/wiki/help/en/h​owto/openstack#Authenticationwithswauth
17:39 glusterbot Title: help/en/howto/openstack – Cyberduck (at trac.cyberduck.ch)
17:40 cmcdermott1 semiosis: I would prefer to have them named the same, but I wasn't able to get that to work - I deleted and re-created the directories and tried different directory names, I even removed and re-installed gluster
17:40 semiosis did you check the ,,(extended attributes) on parent directories up to / ?
17:40 glusterbot (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://hekafs.org/index.php/2011/​04/glusterfs-extended-attributes/
17:40 semiosis up to and including /
17:41 cmcdermott1 no just up to /gluster
17:41 semiosis check / as well, as hagarth suggested :)
17:41 cmcdermott1 this should do it right:
17:41 cmcdermott1 [chris@prod01 ~]$ sudo getfattr -d -m ".*" /
17:41 cmcdermott1 [chris@prod01 ~]$
17:41 atrius elyograg: UFO?
17:42 elyograg atrius: unified file and object store.
17:42 semiosis cmcdermott1: glusterbot just gave you the command above...  getfattr -m .  -d -e hex {filename}
17:42 atrius elyograg: ah, thanks
17:42 cmcdermott1 [chris@prod01 /]$ sudo getfattr -m . -d -e hex /
17:42 cmcdermott1 [chris@prod01 /]$
17:43 joeto joined #gluster
17:43 cmcdermott1 [chris@prod01 /]$ sudo getfattr -m . -d -e hex /*
17:43 cmcdermott1 [chris@prod01 /]$
17:43 JoeJulian tc00per: possibly is split-brain (which is what I think you were about to ask, but it got cut off) or could be something else. If you search the bricks in the subdirectory you're trying to remove, there may be files that don't show up in an ls on the client. With those specific files, see if you can "stat $file" on the client. If they show up, that'll probably cure the problem and you can then delete them.
17:43 cmcdermott1 not sure if {filename} should be /, but one of those should have been right
17:43 semiosis cmcdermott1: could you check that on prod02 as well?  no need to copy the result here, if there's none.
17:44 semiosis cmcdermott1: yeah {filename} should be /
17:44 cmcdermott1 same thing - no results
17:44 semiosis hm
17:44 JoeJulian cmcdermott1: I think you'll find that one one server you'll have prod01 and prod02 and that server will have all the files.
17:45 JoeJulian s/prod0/brick/
17:45 glusterbot What JoeJulian meant to say was: cmcdermott1: I think you'll find that one one server you'll have brick1 and prod02 and that server will have all the files.
17:45 semiosis i know you said peer status says everything is ok, but i still think you should check to make sure that names "prod01" and "prod02" resolve to IP correctly on both machines
17:46 semiosis also that the UUIDs in /var/lib/glusterd/glusterd.info (iirc) are different on both
17:46 cmcdermott1 I added hosts entries on both servers - the prod01 and prod02 hostnames resolve to each server's local IP
17:46 semiosis if those were mixed up it may cause this sort of problem
17:47 cmcdermott1 AHA! That's it - they have the same UUID
17:47 semiosis woo!
17:47 cmcdermott1 I guess it didn't get re-generated when I reinstalled
17:47 cmcdermott1 how do I generate new UUIDs?
17:48 semiosis well glusterd generates it if it's missing when starting up
17:48 cmcdermott1 ok so just delete that file and restart the service... easy enough
17:49 semiosis idk how you could have probed between these machines though
17:49 tc00per left #gluster
17:49 semiosis you may need to fix the UUIDs recorded for the peers in /var/lib/glusterd/peers
17:50 semiosis or just start all over again with empty /var/lib/glusterd directories on both servers, that's probably easiest if you can do it
17:51 ekuric1 joined #gluster
17:54 cmcdermott1 awesome - I took your advice semiosis, and removed the entire /var/lib/glusterd directory on both hosts (after stopping and deleting the volume)
17:54 semiosis cool, one more thing to note is that you should remove the brick directories that you created earlier
17:54 cmcdermott1 I restarted the glusterd service and they both got unique uuid's and I was able to successfully create and start a new volume with matching brick directories
17:54 cmcdermott1 yeah I did that too
17:54 cmcdermott1 thank you so much for your help!
17:54 semiosis ok then you should be all set to start over
17:54 semiosis yw!
17:56 elyograg if my xfs volume is /bricks/b1 but I ask gluster to use /bricks/b1/vol, will the xattrs be on the xfs root or the vol directory?  instinct says the latter, but I'd just like to be sure.
17:56 t35t0r joined #gluster
17:56 t35t0r joined #gluster
17:57 JoeJulian you are correct. "vol" will be the brick root and will be the first directory with xattrs.
17:58 hattenator joined #gluster
17:59 elyograg JoeJulian: thanks.  more reason to use a directory within the filesystem for the bricks.  my primary reason for it is to ensure that gluster won't kill the server by filling up the root fs.
17:59 JoeJulian Ooh, you just gave me a good idea.
18:01 elyograg I did notice that if that directory doesn't already exist, gluster volume create will go ahead and make it.  although that is convenient, I think it might be a good idea to consider it a bug.  if you fat fingered something, you could end up with gluster using a location you're not expecting.
18:02 elyograg I don't know if it will make an entire path that doesn't exist, but if the last dir doesn't exist, I know that will be created.
18:04 tc00per joined #gluster
18:04 semiosis elyograg: +1 i'd also prefer an error to a new dir being created
18:04 semiosis also a warning if that dir is on the same device as /
18:04 semiosis elyograg: could you file a bug please?
18:04 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
18:04 semiosis :D
18:04 elyograg i can file the bug, unless you're already there.
18:04 elyograg ;)
18:05 semiosis no i'm busy working on a bash script to produce XLSX :(
18:05 elyograg would that be glusterd, or a different component?
18:05 semiosis i think glusterd and if we're wrong they'll change it in triage
18:15 themadcanudist joined #gluster
18:15 themadcanudist left #gluster
18:16 tc00per JoeJulian: My irc client lost it's mind... didn't see your reply. I'll bring you back up to speed.
18:16 tc00per Tried to copy OVER TOP of the problem tree. Got access denied errors while trying to write the file contents to the two problem dirs.
18:17 tc00per One one of the peers one of the problem dirs exists and contains files. Stat'ing the dir and/or files doesn't make them readable on client. Directory permissions are... Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
18:19 tc00per Same directory or source has perms... Access: (0775/drwxrwxr-x)  Uid: ( 9153/ tcooper)   Gid: ( 9153/ tcooper)
18:19 tc00per s/One one/On one/
18:19 glusterbot What tc00per meant to say was: On one of the peers one of the problem dirs exists and contains files. Stat'ing the dir and/or files doesn't make them readable on client. Directory permissions are... Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
18:20 glusterbot New news from newglusterbugs: [Bug 864611] When brick directory doesn't exist, volume creation should fail <https://bugzilla.redhat.com/show_bug.cgi?id=864611>
18:20 wushudoin joined #gluster
18:21 JoeJulian That directory only exists on one peer?
18:24 tc00per No... directory exists on all peers/all bricks with different sizes/dates
18:24 tc00per Both problem directories do
18:24 JoeJulian What's the error in the client log show?
18:29 tc00per grep st001_ser0005 /var/log/glusterfs/glusterfs.log.... tail end of output.... http://fpaste.org/os49/
18:29 glusterbot Title: Viewing client log no subdir access by tc00per (at fpaste.org)
18:30 tc00per st001_ser0005 is the directory, not a file.
18:30 JoeJulian Inside that directory, is there a symlink?
18:33 redsolar joined #gluster
18:33 tc00per On one of the peers there are 'entries' inside that directory. Here is a 'typical' example... ---------T. 2 root root 0 Oct  9 08:50 im1631.dcm. They should be/(were) symlinks to files in a parallel branch of the tree.
18:34 tc00per From the client perspective.
18:34 tc00per Not sure what actually happens in glusterfs... that is why I'm testing... :)
18:36 JoeJulian That's a sticky pointer that you shouldn't be seeing from the client. Use the ,,(split-brain) instructions to remove those files from the bricks.
18:36 glusterbot (#1) learn how to cause split-brain here: http://goo.gl/nywzC, or (#2) To heal split-brain in 3.3, see http://joejulian.name/blog/fixin​g-split-brain-with-glusterfs-33/ .
18:36 JoeJulian #2
18:36 atrius 3.3 hasn't been released as yet, correct? about when is it expected?
18:36 JoeJulian atrius: Which distro do you prefer?
18:37 tc00per I don't see the above from the client... only on the peer. Client cannot enter the directory.
18:37 atrius JoeJulian: currently Ubuntu
18:37 JoeJulian @ppa repo
18:37 glusterbot JoeJulian: semiosis' Launchpad PPAs have 32 & 64-bit binary packages of the latest Glusterfs for Ubuntu Lucid - Precise. http://goo.gl/DzomL (for 3.1.x) and http://goo.gl/TNN6N (for 3.2.x) and http://goo.gl/TISkP (for 3.3.x). See also @upstart
18:37 atrius oh, sorry.. 3.3 is here... my mistake :D
18:37 JoeJulian tc00per: Strange, that should have been removed with the files. I wonder if I can repro that...
18:37 atrius JoeJulian: thanks :)
18:38 JoeJulian tc00per: Still, use the same instructions for removing those from the bricks.
18:38 JoeJulian atrius: you're welcome. :)
18:38 kkeithley_wfh yes, 3.3.0 was released in April or May. hagarth indicated on the gluster-devel mailing list earlier today that 3.3.1 will be out later this week.
18:39 JoeJulian I need to get organized. I have a bug fix to test that didn't apply cleanly, but I was able to work around it. Need to build it, now, so I can install and test.
18:40 atrius if the bricks are provided via iSCSI... is is the idea that you'd format the iscsi device... mount it on all machines, and then reshare/manage it via gluster?
18:40 semiosis atrius: no way
18:40 * JoeJulian boggles
18:41 kkeithley_wfh mount it on all machines? I hope you're using a cluster file system like ocfs2 for that
18:41 * glusterbot boggles as well.
18:41 atrius lol
18:41 atrius okay... each machine with its own iscsi share, format, mount, gluster?
18:41 semiosis i dated a girl who liked to boggle
18:41 semiosis http://en.wikipedia.org/wiki/Boggle
18:41 glusterbot Title: Boggle - Wikipedia, the free encyclopedia (at en.wikipedia.org)
18:41 JoeJulian Layer some clustering on top of clustering. That sounds... interesting...
18:42 atrius JoeJulian: it sounded stupid.. but for some reason that's what popped in my mind
18:43 semiosis JoeJulian: i saw a Q on CGO about running mongodb over glusterfs... sorry but i had to downvote your answer ;)
18:43 atrius i imagine my second idea is full of less fail?
18:43 JoeJulian much, yes. :D
18:43 atrius lol
18:43 semiosis atrius: why use iscsi at all?
18:43 JoeJulian Though, of course, that iscsi device is still a spof.
18:43 semiosis atrius: the huge benefit of glusterfs is that you dont need any of that
18:43 semiosis just regular local storage
18:44 semiosis (never mind that i'm actually using EBS, forget about that)
18:44 JoeJulian lol
18:44 atrius semiosis: mainly because i'm trying to simulate what our san device will likely provide in production... i'm figuring it'll either end up iscsi or something i can't simulate here anyway
18:44 atrius JoeJulian: true, still a SPOF
18:44 semiosis glusterfs > san
18:44 kkeithley_wfh That iSCSI device could be doing RAID[56] with a battery backed cache. If you've got money for that kind of hardware...
18:44 tc00per Will do... looking at 'how to cause a split-brain' I would like to state clearly my test case in case you want to try to replicate.
18:44 * semiosis goes to lunch
18:44 semiosis bbiab
18:45 bulde joined #gluster
18:45 tc00per I have a test directory on my workstation with ~19K files/links/dirs of various sizes. Typical for our workflow. A lot of symlinks exist, either to files adjacent inside a directory OR to files in another part of the tree.
18:45 tc00per I create the gvol, copy data in, find all files in the tree, then rm -rf the tree repeatedly collecting times. This worked flawlessly with my original 1 x 2 repl. glusterfs. I have various other failures after migrating 1 x 2 repl to dist-repl of multiple sizes.
18:46 JoeJulian tc00per: There's a self-heal bug when symlinks exist in a directory that needs healed. It'll be fixed in 3.3.1 I believe.
18:46 tc00per Problems seem to be associated with symlinks.
18:47 tc00per I'll be sure to test it... dead in the water without it being fixed. :(
18:47 atrius kkeithley_wfh: true
18:47 tc00per On to 'fixing split-brain'...
18:47 kkeithley_wfh I need to set up krb5 auth for a couple machines to play with NFS auth. Anyone have experience with that? I don't really want to set up FreeIPA or OpenLDAP for ids, I just want to use ids in /etc/passwd.  Got a pointer to a good how-to?
18:48 kkeithley_wfh s/Got a pointer/Anyone got a pointer/
18:48 glusterbot What kkeithley_wfh meant to say was: I need to set up krb5 auth for a couple machines to play with NFS auth. Anyone have experience with that? I don't really want to set up FreeIPA or OpenLDAP for ids, I just want to use ids in /etc/passwd.  Anyone got a pointer to a good how-to?
18:48 atrius of course, could one not just have N iscsi links (or local drives, whatever) on each node from N iscsi servers and thus eliminate that spof?
18:48 pdurbin kkeithley_wfh: mmm, kerberos, nfs. are you thinking about home directories? :)
18:49 kkeithley_wfh pdurbin: yup, basically.
18:50 tc00per JoeJulian: Can't fix what's not broke...:) gluster volume heal VOLNAME info split-brain says nothing is marked split-brain.... :(
18:50 tc00per s/:)/:(/
18:50 pdurbin kkeithley_wfh: nice! my problem! :) http://irclog.perlgeek.de/g​luster/2012-10-05#i_6038531
18:50 glusterbot tc00per: Error: '/^(?!s([^A-Za-z0-9\\}\\]\\)\\>\\{\​\[\\(\\<\\\\]).+\\1.+[ig]*).*:).*/' is not a valid regular expression.
18:50 glusterbot Title: IRC log for #gluster, 2012-10-05 (at irclog.perlgeek.de)
18:50 JoeJulian Right, it's not technically, but you still have to delete both the file and the gfid file from .glusterfs.
18:50 atrius also.. if gluster doesn't support "structured data" like databases, does that mean you couldn't use it for VM backing storage which contains a database? or is that line only referring to raw access?
18:50 pdurbin glusterbot: you crazy
18:51 kkeithley_wfh pdurbin: indeed.
18:51 JoeJulian Ah, it's the parenthesis.
18:52 sshaaf joined #gluster
18:52 JoeJulian atrius: I run mysql on it.
18:53 atrius JoeJulian: okay... so that note in the "what is gluster" thing is more cautionary and less "we're all going to die and you're going to implode the universe!!"
18:53 JoeJulian As long as you don't cross the streams.
18:53 johnmark haha
18:53 cmcdermott1 left #gluster
18:53 JoeJulian It may not handle millions of selects / second though.
18:54 atrius JoeJulian: clearly not... also you'd want to monitor the neutrino and tetrion fields
18:55 atrius JoeJulian: fair enough... i don't think we're doing that kind of volume anyway... we'll likely put the active DB in different storage and just mount that inside the VM...
18:55 * JoeJulian mounts a gluster volume that houses my mysql data from inside the vm.
18:56 atrius we're probably going to end up with a small SAN in our environment whether I want one or not... so i may as well make it work for its electronic bread
18:58 abyss^ joined #gluster
18:59 tc00per JoeJulian: re: fixing split-brain. If the problem dir exists on all peers/bricks must I rinse-repeat the removal steps EVERYWHERE?
18:59 johnmark JoeJulian: I mis-read that
18:59 johnmark thought you said "hoses" instead of "houses" and was wondering why you weren't cursing
19:00 r__2 joined #gluster
19:02 r__2 hey, can anyone explain what "Skipping entry self-heal because of gfid absence" means?
19:03 red_solar joined #gluster
19:12 camel1cz joined #gluster
19:17 H__ don't know what that means but it sounds scary
19:17 H__ i hope someone in the know will help out
19:18 r__2 thanks H__
19:18 jdarcy It means that we found files that don't seem to have been put there by GlusterFS itself (e.g. written locally into the brick directory).
19:20 r__2 thank you jdarcy, that makes sense.  I'll upload via the client.  unless anyone knows a way to force gluster to recognize the file/add a gfid after manual addition to a brick?
19:21 semiosis stat its absolute path through the client
19:21 semiosis though of course that may not always work, it's best to always write through the client
19:22 r__2 thanks semiosis, i'll try the absolute path and upload from client in the future
19:25 semiosis yw
19:26 tc00per JoeJulian: What is the bug number for the symlink bug you mentioned earlier today?
19:29 JoeJulian bug 831151
19:29 glusterbot Bug https://bugzilla.redhat.com​:443/show_bug.cgi?id=831151 unspecified, unspecified, ---, pkarampu, ON_QA , Self heal fails on directories with symlinks
19:35 elyograg does gluster have the concept of global properties, or are they all on the volume?
19:35 andreask joined #gluster
19:36 JoeJulian All per-volume.
19:38 elyograg ok.  the reason I ask is because I am working a bug filing that semiosis mentioned earlier.  specifically, a warning when a brick is on the root filesystem.  it occurred to me that it might be better to have a setting that controls whether volumes can be created with bricks on a root filesystem.
19:40 aliguori joined #gluster
19:40 elyograg just now I had an alternate idea.  You allow the creation to happen, with a warning message.  then unless an "allow.rootfs" (or similar) volume property is enabled, it won't be possible to *start* a volume that has bricks on root filesystems.
19:45 kkeithley_wfh the root file system or the top-level-directory of some file system?
19:46 tc00per I can see the '6 month release cycle' information for 3.4 on the community website. What is 'typical' release schedule for 'dot' releases? We're getting close to 6 months since 3.3.0 was released if you go by the date stamp on the release files.
19:46 kaisersoce joined #gluster
19:47 elyograg kkeithley_wfh: the aim really is to prevent "didn't mean that directory" or "forgot to mount the disk" situations from filling up the rootfs.
19:47 semiosis +1
19:48 elyograg which is why I would suggest a property, so if you really do want to allow the brick to exist on the root fs, which might be very common in testbeds, it'll still be possible.
19:48 semiosis kkeithley_wfh: to catch you up, the problem elyograg identified is that if you typo a brick directory it will create that directory
19:48 semiosis rather than warning you that it doesn't exist
19:48 JoeJulian I still like the option of --yes-i-know-what-im-doing to override any sanity check.
19:49 kkeithley_wfh tc00per: dot releases are on ~six month cycles. hagarth posted on gluster-devel mailing list earlier today that 3.3.1 will be out this week.
19:49 kkeithley_wfh semiosis: yup
19:49 tc00per Cool... what do you call the 3.3.1 release if not a 'dot release'? Time to subscribe to the devel mailling list... :)
19:50 JoeJulian I wonder if there's a way to get bugfix releases on a shorter cycle. It seems unreasonable to have to wait 6 months for what should have been showstoppers if they'd been found.
19:50 semiosis afaik version numbers are calle <major>.<minor>.<patch>
19:50 semiosis so i'd call it a patch release
19:50 JoeJulian yeah, sounds good.
19:50 kkeithley_wfh tc00per: we're saying the same thing. 3.3.1 is the dot release, a little sooner than six months in this case
19:51 r__2 left #gluster
19:51 kkeithley_wfh major/minor/teeny
19:51 JoeJulian major/minor/oops
19:52 kkeithley_wfh I'd say the -{5,7,8,9,11}, some of which have patches for crash bugs, are patch releases. Meh, what's in a name.
19:52 kkeithley_wfh er, the -{5,7,8,9,11} in my fp.o repo
19:53 kkeithley_wfh clearly the coffee earlier didn't help
19:53 kkeithley_wfh the 3.3.0-{5,7,8,9,11} RPMs in my fp.o repo
19:54 kkeithley_wfh Although those are just me rolling my own, they aren't official in any way.
19:54 kkeithley_wfh and my brain is fried by trying to set up a krb5 kdc.
19:55 tc00per kkeithley: how long after the 'teeny' release should we expect to see the update in your repo?
19:55 johnmark JoeJulian: agreed. that's something we should address after this release
19:56 johnmark kkeithley_wfh: FYI your blog account should be fine. I looked at it last night, and I didn't see anything obvious to prevent you from posting or anything
20:00 elyograg are volumes marked in some way that gluster can detect which version was last running the volume?
20:01 elyograg or more specifically, are bricks marked like that?
20:03 tc00per Since I'm sunk testing until 3.3.1 I'm willing to try applying the qa3 rpms to my cluster systems to see if my symlink problem gets solved. Any thoughts/comments on doing this? OS is CentOS 6.3.
20:09 Daxxial_1 joined #gluster
20:10 jbrooks joined #gluster
20:15 hattenator tc00oer: I don't think there's much of a trick to it.  I did that upgrade a week or so ago.  I'm not sure if you should run both versions on different bricks simultaneously, though.  It's probably better to umount it and upgrade every brick and every client together.  But I ran different versions simultaneously, and although my cluster was already split-brained and broken, it didn't get any more broken.  Just rpm -U and you shoul
20:18 lanning joined #gluster
20:21 glusterbot New news from newglusterbugs: [Bug 864638] create volume property to allow/disallow starting a volume with bricks on the root filesystem <https://bugzilla.redhat.com/show_bug.cgi?id=864638>
20:21 tc00per Looks like there are some strange (old?) dependencies in the rpm's at http://bits.gluster.com/pub/glu​ster/glusterfs/3.3.1qa3/x86_64/
20:21 glusterbot Title: Index of /pub/gluster/glusterfs/3.3.1qa3/x86_64 (at bits.gluster.com)
20:21 tc00per Want openssl098e-0.9.8e-17.el6.centos.2.i686 and compat-readline5-5.2-17.1.el6.x86_64
20:22 gbrand_ joined #gluster
20:23 elyograg I can't be sure because my memory is crap, but isn't that release of openssl vulnerable to some of the more nasty attacks?
20:23 tc00per Have...openssl.x86_64            1.0.0-25.el6_3.1           @updates
20:23 tc00per readline.x86_64           6.0-4.el6                  @base/$releasever
20:24 tc00per openssl updated on my server on Sep 26, 2012... the day I installed it.
20:27 puebele joined #gluster
20:57 neofob joined #gluster
20:59 kkeithley_wfh johnmark: yes, today I could see all my posts.
21:08 sashko joined #gluster
21:13 Guest21882 joined #gluster
21:14 badone joined #gluster
21:14 Guest21882 hi all, trying to enable posix locking for my two node gluster replicated cluster. Can anyone give me some pointers ? I cant figure out from documentation
21:14 badone_home joined #gluster
21:18 semiosis Guest21882: it's enabled by default when using the glusterfs native FUSE client
21:22 Guest21882 semiosis: ok, I am having some issues with ctdb which requires this, so was looking at this
21:24 Guest21882 semiosis: is this feature enable by default in all versions ?
21:24 badone_home joined #gluster
21:24 semiosis idk about "all" but afaik yes since 3.0, probably earlier too
21:26 Nr18 joined #gluster
21:26 Guest21882 semiosis: do you know ctdb and use with gluster ?
21:27 semiosis no
21:27 semiosis @lucky ctdb
21:27 glusterbot semiosis: http://ctdb.samba.org/
21:27 semiosis that?
21:27 Guest21882 semiosis: yes :)
21:28 semiosis and it wants to use a posix lock on a file to pick the active server?
21:28 semiosis sounds like a bad idea to me
21:29 johnmark hrm... we have a doc somewhere about ctdb + glusterfs
21:29 semiosis @lucky ctdb glusterfs
21:29 glusterbot semiosis: http://download.gluster.com/pub/gluster/sys​tems-engineering/Gluster_CTDB_setup.v1.pdf
21:29 semiosis that?
21:29 * semiosis is on a roll!
21:29 Guest21882 have read it a few times :)
21:30 johnmark Guest21882: this one? http://download.gluster.com/pub/gluster/sys​tems-engineering/Gluster_CTDB_setup.v1.pdf
21:30 johnmark Guest21882: well, we intend to update it over the next couple of weeks
21:30 Guest21882 semiosis/johnmark: having a discussion with the people at @samba, have read the doc yes
21:30 johnmark doh...
21:30 johnmark semiosis: I'm a bit slow :)
21:30 * johnmark will remember @lucky
21:31 semiosis hehe
21:31 johnmark @lucky semiosis
21:31 glusterbot johnmark: http://en.wikipedia.org/wiki/Semiosis
21:31 johnmark lulz
21:31 Guest21882 johnmark: I cannot get ctdb to use the lock file on the shared volume (glusterfs), it complains it cannot find it
21:31 semiosis johnmark: hahaha
21:31 semiosis "Semiosis is any form of activity, conduct, or process that involves signs, including the production of meaning."
21:32 johnmark Guest21882: I don't know what to tell you, unfortunately
21:33 Guest21882 johnmark: so I guess there is now magic code to put in my config files :) .... why cant things just work :)
21:34 johnmark Guest21882: *sigh* I hear you
21:35 johnmark Guest21882: did you see this? http://www.mentby.com/Group/samba-technical/​ctdb-relock-file-issues-with-glusterfs.html
21:35 glusterbot Title: ctdb relock file issues with glusterfs - Samba Technical (Internals) (at www.mentby.com)
21:35 Guest21882 anyone used some kind of similar tool to make cluster solutions for nfs, samba or other stuff together with gluster ?
21:35 johnmark that sounds like your problem
21:36 semiosis Guest21882: could you please pastie the last ~20 lines from your glusterfs client log file?
21:36 semiosis pastie.org or similar
21:37 Guest21882 johnmark: yes, I am on the mailling list :) .. not sure if it is the same, but very similar
21:37 tryggvil_ joined #gluster
21:38 semiosis Guest21882: last ~20 lines assuming this problem is ongoing, otherwise lines from around the time when you last reproduced it
21:39 Guest21882 http://pastie.org/5026490
21:39 glusterbot Title: #5026490 - Pastie (at pastie.org)
21:40 semiosis Guest21882: that doesn't look like a log
21:40 semiosis client log file is usually /var/log/glusterfs/client-mount-point.log
21:41 Azrael808 joined #gluster
21:41 Guest21882 sorry...
21:42 Guest21882 please hold... trouble ...
21:46 redsolar joined #gluster
21:47 Guest21882 semiosis: here we go: http://pastie.org/5026525
21:48 glusterbot Title: #5026525 - Pastie (at pastie.org)
22:24 neofob left #gluster
23:07 tryggvil joined #gluster
23:10 kaisersoce joined #gluster
23:18 puebele joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary