Camelia, the Perl 6 bug

IRC log for #gluster, 2013-08-09

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:17 awheeler joined #gluster
00:31 poptix joined #gluster
00:34 poptix is there any clean way to permanently remove a node?
00:34 poptix where the data contained therein is automatically rebalanced
00:35 poptix or is it a matter of keeping replication count high enough to deal with multiple node losses, removing the node(s) in question and starting a heal
00:35 a2 remove-brick does exactly that
00:36 poptix that has a scary warning about possible data loss =)
00:37 a2 not the remove-brick start
00:37 poptix I guess the difference I'm looking for is 'remove this node when convenient after a rebalance/heal' vs. 'remove this node NOW and rebalance/heal and hope another node doesn't go down'
00:37 a2 remove-brick start drains data out of the pair to the remaining nodes, and when data is drained out, you remove-brick commit to take it out of config
00:38 poptix ah, great
00:38 poptix can you tell me in what instances remove-brick can cause data loss? the manual doesn't elaborate
00:39 poptix from http://www.gluster.org/wp-content/up​loads/2012/05/Gluster_File_System-3.​3.0-Administration_Guide-en-US.pdf
00:39 glusterbot <http://goo.gl/bzF5B> (at www.gluster.org)
00:39 a2 just remove-brick (without "start") forces a brick out of config - really meant to remove a node after it is dead
00:39 poptix # gluster volume remove-brick test-volume server2:/exp2 start
00:39 poptix Removing brick(s) can result in data loss. Do you want to Continue? (y/n)
00:39 poptix oh
00:39 poptix jeez, got it
00:39 a2 eh, that's add.. it shouldn't say "data loss" when you say "start". that's misleading
00:39 a2 *odd
00:39 joshit_ joined #gluster
00:40 asias joined #gluster
00:40 joshit_ has anyone placed their mail dir within a gluster with great performance?
00:40 poptix it's at the top of page 41, perhaps it's outdated
00:40 joshit_ or tweaked gluster to make mail dir perform great?
00:41 a2 joshit_, using a recent enough kernel with readdirplus support in FUSE should help maildir performance on gluster
00:42 joshit_ will read up on it, thanks
00:43 a2 you will need 3.4 glusterfs to accompany
00:44 johnsonetti joined #gluster
00:44 joshit_ running this also so will read :)
00:54 B21956 left #gluster
01:06 yinyin joined #gluster
01:10 bala joined #gluster
01:59 thomasle_ joined #gluster
02:06 _pol joined #gluster
02:07 _pol_ joined #gluster
02:12 ultrabizweb joined #gluster
02:44 bala joined #gluster
03:14 badone_ joined #gluster
03:33 recidive joined #gluster
03:37 yinyin joined #gluster
04:17 bala joined #gluster
04:37 yinyin joined #gluster
05:41 hagarth joined #gluster
05:42 joshit_ joined #gluster
05:44 harish joined #gluster
05:44 joshit_ can anyone help me? need to set max-file-size and cache-timout for performance quick read on a certain volume
05:44 joshit_ whats the correct commands as what im typing keeps asking me did i mean this, did i mean that
05:45 joshit_ we enabled first part with > gluster volume set VOLNAME performance.quick-read on
06:11 joshit_ i'm trying to > gluster volume set volumename performance.quickread.cache-timeout <1>
06:11 joshit_ not working >.<
06:13 bulde joined #gluster
06:18 joshit_ does anyone know how to set performance.quick-read cache-timeout?
06:26 harish joined #gluster
06:28 wgao joined #gluster
06:47 jiku joined #gluster
06:49 vimal joined #gluster
06:51 bala joined #gluster
06:56 puebele joined #gluster
07:03 17WAB99DI joined #gluster
07:06 ricky-ticky joined #gluster
07:12 andreask joined #gluster
07:14 ekuric joined #gluster
07:29 rgustafs joined #gluster
07:47 mooperd joined #gluster
08:08 mbukatov joined #gluster
08:09 bala joined #gluster
08:14 mbukatov joined #gluster
08:20 tjikkun_work joined #gluster
08:29 X3NQ joined #gluster
08:33 bala joined #gluster
08:34 samsamm joined #gluster
08:34 samsamm hi all
08:35 samsamm anyone explain this line to me from a glusterfs tutorial?
08:36 samsamm [ec2-user@ip-172-31-11-2 ~]$ mkfs.ext4 -m 1 -L gluster /dev/sdg
08:36 samsamm mke2fs 1.41.12 (17-May-2010)
08:36 samsamm Could not stat /dev/sdg --- No such file or directory
08:36 samsamm The device apparently does not exist; did you specify it correctly?
08:36 samsamm cant get it to work
08:36 ndevos samsamm: what glusterfs tutorial would that be? the advice is to use xfs...
08:38 samsamm http://www.gluster.org/category/aws-en/
08:38 glusterbot Title: GlusterFS in AWS | Gluster Community Website (at www.gluster.org)
08:38 samsamm this oen
08:41 stickyboy What would cause a host to show up using its IP in `gluster peer status`?  ie, Hostname: 192.168.5.27   (rather than its hostname).
08:42 ndevos samsamm: I suggest you use the 'getting started' or 'quick guides' from http://gluster.org/community/doc​umentation//index.php/Main_Page
08:42 glusterbot <http://goo.gl/E3Jis> (at gluster.org)
08:42 samsamm ndevos: ok thanks. ill have a read
08:43 ndevos stickyboy: mostly that happens when the peer probe was done by IP, and not by hostname
08:43 stickyboy ndevos: Ok.
08:43 stickyboy I'm not particularly worried... but meh.
08:43 stickyboy On one host it shows the hostname, on another it shows the IP.
08:44 ndevos ~hostnames | stickyboy
08:44 glusterbot stickyboy: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
08:44 stickyboy Ok.  Lemme process that, hehe.
08:45 NuxRo the bigger your setup grows the funnier it gets :D
08:45 NuxRo ansible or puppet might come in handy
08:45 stickyboy Ah, I just peer probed it from the new host and it fixed the name.  Great.
08:45 stickyboy NuxRo: Yah, I've been playing with ansible all week actually... really starting to like it.
08:45 stickyboy I figured out how to deploy my fuse mounts via ansible, that is cool.
08:46 stickyboy ... and now I need to redeploy my firewall rules to account for new bricks...
08:47 stickyboy I templated my /etc/sysconfig/iptables with Ansible / jinja2 yesterday.  :D
08:48 NuxRo nice nice
08:49 stickyboy I have one group variable for my storage servers:   gluster_brick_ports: 24009:24012
08:49 partner hmm any idea why on my 3x2 distributed-replicated volume the disk usage between the replica partners varies a lot?
08:49 stickyboy And that goes straight into iptables :D
08:49 partner /dev/sdc               20G   20G   76K 100% /export/brick2
08:49 partner /dev/sdc               20G   19G  1.1G  95% /export/brick2
08:50 partner yeah, its full, its testing instance, gluster in some limbo due to that?
08:50 bulde joined #gluster
08:51 ndevos stickyboy: care to blog about that config/setup? ping johnmark the url and it can get included in the gluster.org blogs
08:51 partner no active sinks for performing self-heal on file <gfid:a6052b0d-ffc7-4ab6-89d2-046125980608>
08:52 partner ah
08:52 partner 0-rv0-replicate-1: Stopping crawl for rv0-client-2 , subvol went down
08:54 partner procs are all up and kinda seems to work i guess
08:54 partner 0-rv0-replicate-1: Stopping crawl as < 2 children are up
08:55 mohankumar joined #gluster
08:56 stickyboy ndevos: Actually I'm already syndicated on gluster.org, and johnmark already guilt trips me on twitter :D
08:56 stickyboy ndevos: I will def blog about it.
08:56 ndevos stickyboy: great!
08:58 neuroticimbecile ndevos: where's your blog?
08:58 bala joined #gluster
08:58 ndevos neuroticimbecile: blog.nixpanic.net
08:59 ndevos neuroticimbecile: http://blog.nixpanic.net/search/label/Gluster for the gluster bits
08:59 glusterbot Title: Nixpanic's Blog: Gluster (at blog.nixpanic.net)
09:02 stickyboy Mine's http://mjanja.co.ke/
09:02 glusterbot Title: Mjanja Tech: Ujanja ni uhai (hustling is life) (at mjanja.co.ke)
09:03 neuroticimbecile got it. thanks guys.
09:04 ndevos stickyboy: is that a github hosted blog, or you added the 'fork me' yourself?
09:05 stickyboy ndevos: It's a WordPress plugin
09:05 ndevos stickyboy: ah, ok
09:05 stickyboy So "me" is more referring to me as a person, not the blog hehee.
09:08 NuxRo so is the message intended for the girls?
09:09 samsamm could anyone explain this to me? "Assuming you have a brick at /dev/sdb: "
09:09 stickyboy NuxRo: :P
09:09 samsamm there is nothing at /dev/sdb
09:10 stickyboy samsamm: Well /dev/sdb is just a disk.  It could be /dev/sdc on your system, depending on how many you have and how they are enumerated.
09:10 NuxRo samsamm: where is that from?
09:10 samsamm http://gluster.org/community/documentat​ion/index.php/Getting_started_configure
09:10 glusterbot <http://goo.gl/FSWvOc> (at gluster.org)
09:10 stickyboy /dev/sda is the first-discovered SATA disk on the system, sdb is second, sdc is third, etc.
09:10 samsamm ah i see!
09:11 samsamm so i should just change that to whatever my ebs disk is?
09:11 stickyboy That's just generic Linux stuff, so you gotta adapt it to your system.
09:11 NuxRo samsamm: yep
09:11 samsamm so im guessing it would be /dev/sda1 on my system
09:11 NuxRo a brick can be any directory in your syste, be it a specifically mounted disk or not
09:12 NuxRo isn't sda1 your / partition?
09:12 samsamm yeah
09:12 samsamm so i could just use any directory?
09:12 stickyboy samsamm: That's fine for testing, but not very practical.
09:12 stickyboy ie, / will get full and crash your system if you use it in production ;)
09:13 samsamm yeah very true.
09:13 partner uuh, no build for squeeze for 3.3.2 :/
09:13 samsamm now i understand what its using it for though
09:14 NuxRo for production i use a raid6 mounted under /bricks and then I use some like: volume create BLAH replica 2 server1:/bricks/BLAH server2:/bricks/BLAH
09:15 morse joined #gluster
09:15 samsamm so i could potential just create another ebs volume and attach it to the instance and use that?
09:15 NuxRo of course
09:15 NuxRo it wold be better practice
09:15 samsamm ok that seems good to me
09:16 NuxRo good luck :)
09:16 samsamm im basically experimenting with gluster fs to overcome the fact that s3 is slowwww
09:17 NuxRo what do you want to use it for? gluster can be slow, too
09:17 stickyboy Truth
09:18 samsamm we need a scalable storage solution which will be accessible to all of our ec2 instances
09:19 samsamm it needs to be relatively fast
09:19 NuxRo samsamm: gluster fits this role
09:19 NuxRo i use a volume for a customer with many servers, he mounts it via nfs and keep shared stuff there
09:21 samsamm how does high availability work with gluster? for example if one of the bricks died, would some of the data be inaccessible?
09:22 samsamm or could we do something like pair-replication so we could have 3 pairs (6bricks). which would alleviate any issues.
09:23 NuxRo samsamm: you must use a replicated volume for that, and yes, you can have pairs
09:23 NuxRo pairs of 2, of 3, of however many you want
09:23 NuxRo but more than 3 latency will become a real problem imho
09:23 samsamm yeah, that would be relatively robust i think.
09:24 NuxRo http://joejulian.name/blog​/dht-misses-are-expensive/
09:24 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
09:24 stickyboy Everyone's putting gluster in AWS... makes me wonder what kind of storage Amazon uses behind the scenes ;)
09:24 NuxRo i recommend you read julian's blog, there are many things in there you should know
09:25 NuxRo samppah: start with pairs of 2 and see how it goes, you can increase it if you need
09:26 samsamm joined #gluster
09:26 samsamm pc just died! :(
09:26 NuxRo ops, i was talking by myself :)
09:26 NuxRo 10:24 < NuxRo> i recommend you read julian's blog, there are many things in there you should know
09:26 NuxRo start with pairs of 2 and see how it goes, you can increase it if you need
09:27 hagarth folks, please add your pet peeve in 3.4.0 to be addressed here: http://www.gluster.org/community/docu​mentation/index.php/Backport_Wishlist
09:27 glusterbot <http://goo.gl/6LCcg> (at www.gluster.org)
09:29 ujjain joined #gluster
09:29 samsamm yeah ill take a read. got plenty of time for proof of concept on this. so will try and dive deep into it
09:31 stickyboy How are gluster ports enumerated?  I just added a distributed volume on a new machine and it got port 24009.  My existing replicated volumes are already 24009:24011, so I figured it would bump to 24012.
09:31 ndevos ~ports | stickyboy
09:31 glusterbot stickyboy: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
09:40 spider_fingers joined #gluster
10:00 edward1 joined #gluster
10:11 purpleidea joined #gluster
10:11 purpleidea joined #gluster
10:18 manik joined #gluster
10:24 stickyboy ndevos: Right, but I have two volumes using the same port.
10:24 stickyboy Are replicated and distributed ports enumerated differently
10:24 stickyboy ?
10:25 ndevos stickyboy: a volume does not have a port, bricks have ports, and a brick does not care to which volume it belongs
10:27 stickyboy ndevos: Ah.
10:28 stickyboy ndevos: Check this out: https://gist.github.com/alanorth/71d9​fddcab30c8b8d8bd/raw/d2fecb1104dccfc3​1b8f71cb507bfa33ca70cf13/gluster.txt
10:28 glusterbot <http://goo.gl/PTsAe1> (at gist.github.com)
10:29 stickyboy Two bricks have 24009.  Different machines of course.
10:31 ndevos stickyboy: yes, that ~ports note above means "Bricks (glusterfsd) use 24009 & up." on one server, different servers have their own personal port-counter
10:33 stickyboy ndevos: Gotcha. :D
10:33 ndevos :)
10:34 stickyboy I'm still scaling beyond 2 machines, so still learning the semantics. :D
10:35 stickyboy btw, "wingu" means "cloud" in Swahili
10:35 stickyboy I thought that was the perfect name for our Gluster army, since we're in Kenya. ;)
10:45 bulde joined #gluster
10:47 ndevos oh, fun, I like that name :)
11:05 jf joined #gluster
11:08 jf hi, trying to work out how a distributed/replicated volume should be mounted on the client side - if I create the client .vol file with remote-host server1 won't that only connect to server1?
11:17 bulde joined #gluster
11:19 cyberbootje hi, by default is gluster binding to a particular ip to listen on?
11:20 cyberbootje in other words, can i safely add ip''s to the gluster machine while everything is running and connect to that ip?
11:22 partner tcp        0      0 *:24007                 *:*                     LISTEN
11:22 partner tcp        0      0 *:24009                 *:*                     LISTEN
11:23 cyberbootje so, no problem there then
11:28 partner "Restarting the rebalance operation causes the rebalance to continue from the point at which it was paused (stopped). "
11:28 partner i'm not exactly sure if that is really happening.. over 1,5M files gone through before first moves happened..
11:30 partner the issue is the rebalance filehandle leak..
11:35 cyberbootje partner: last messages ar not meant for me i suppose?
11:35 partner no :)
11:35 harish joined #gluster
11:35 bulde joined #gluster
11:41 bulde joined #gluster
11:44 kkeithley joined #gluster
11:48 ujjain2 joined #gluster
11:50 tobias- How can clients pull the .vol ? since this is what in use in fstab - is there another way to do it rather than just copy it from a gluster-server?
11:54 jmeeuwen joined #gluster
11:54 manik joined #gluster
11:56 rcheleguini joined #gluster
11:59 stickyboy tobias-: You don't need to mess with the .vol...?
12:00 stickyboy tobias: mount.glusterfs server0:video /export/video/
12:14 jclift_ joined #gluster
12:14 B21956 joined #gluster
12:16 tobias- stickyboy: if server0 is down it wouldn't be able to mount it, right?
12:20 B21956 left #gluster
12:20 kkeithley If you have replication you can mount from any of the nodes with the replicas.  And the server you mount from only needs to be up the first time you mount (again, only true with replication)
12:23 aliguori joined #gluster
12:24 hagarth joined #gluster
12:27 stickyboy tobias-: You can also specify a backup in fstab for mounting purposes:  backupvolfile-server=server1
12:27 stickyboy That's only for the mount, ie when it retrieves the volfile.  After that replication is fine.
12:40 awheeler joined #gluster
12:41 awheeler joined #gluster
12:42 guigui3 joined #gluster
12:43 andreask joined #gluster
12:47 plarsen joined #gluster
13:01 puebele left #gluster
13:08 JoeJulian /*If you have replication*/ you can mount from any of the [servers] /* with the replicas */.  And the server you mount from only needs to be up the first time you mount /*(again, only true with replication */
13:08 JoeJulian The thing is, without replication you can still mount, just some of the files will be missing.
13:08 JoeJulian Unless that's changed with 3.4...
13:09 JoeJulian also, you can mount from any server in the peer group, regardless of whether that peer is even part of the volume you're mounting.
13:12 Durzo joined #gluster
13:14 kkeithley no, none of that has changed. Just trying to keep it simple.
13:15 * JoeJulian laughs at keeping clustered systems simple.
13:16 kkeithley ;-)
13:16 kkeithley gluster is so simple even I can do it!
13:16 JoeJulian hehe
13:17 spider_fingers joined #gluster
13:18 JoeJulian Anybody know which universities are using gluster for gene work? I have someone at the UW (University of Washington) who's shopping for storage.
13:18 andreask joined #gluster
13:18 andreask joined #gluster
13:18 chirino joined #gluster
13:19 JoeJulian johnmark: ^
13:19 kkeithley Harvard is a big user. Not for gene work that I recall, but definitely some big researchy data sets.
13:20 guigui3 joined #gluster
13:24 JoeJulian Have any contact info?
13:28 kkeithley theoretically yes. I'll see if I can resurrect it, otherwise johnmark has it. paging johnmark, please pick up the white courtesy phone.
13:29 JoeJulian "the white courtesy phone" always reminds me of "Airplane".
13:29 JoeJulian "The white phone, dummy!"
13:34 lpabon joined #gluster
13:36 harish joined #gluster
13:37 Twinkies joined #gluster
13:42 m0zes JoeJulian: we are starting genetic work. working on buying 4PB of gluster (possibly + archival tape)
13:44 m0zes we've currently got 576TB or raw storage with gluster on top.
13:46 JoeJulian m0zes: Can I get your email address and put you in touch with my colleague?
13:48 kaptk2 joined #gluster
13:48 m0zes sure. pm'd
13:57 spider_fingers left #gluster
13:59 samsamm [ec2-user@ip-172-31-42-110 dev]$ sudo mkfs.xfs -i size=512 /dev/xvdf
13:59 samsamm sudo: mkfs.xfs: command not found
13:59 samsamm any ideas?
14:00 JoeJulian Assuming an rpm based distro, the way to determine that is: yum provides '*bin/mkfs.xfs'
14:01 samsamm no matches found :(
14:01 * JoeJulian boggles...
14:02 JoeJulian strange since it's in xfsprogs.
14:02 JoeJulian Maybe it's an amazon thing.
14:02 samsamm already tried to install that earlier
14:02 samsamm [ec2-user@ip-172-31-42-110 dev]$ sudo yum install xfsprogs
14:02 samsamm Loaded plugins: amazon-id, rhui-lb, security
14:02 samsamm rhui-eu-west-1-client-config-server-6       | 2.6 kB     00:00
14:02 JoeJulian They do bastardize the distro a bit.
14:02 samsamm rhui-eu-west-1-rhel-server-releases         | 3.7 kB     00:00
14:02 samsamm rhui-eu-west-1-rhel-server-releases-optiona | 3.5 kB     00:00
14:02 samsamm Setting up Install Process
14:02 samsamm No package xfsprogs available.
14:02 samsamm Error: Nothing to do
14:02 JoeJulian @kick samsamm use a pastebin
14:02 samsamm was kicked by glusterbot: use a pastebin
14:03 jclift_ Hmmm, any idea if the CentOS version of it would work?
14:03 samsamm joined #gluster
14:03 jclift_ ... might need to compile your own from SRPM?
14:03 jclift_ Meh... I'll keep out of things.  Should keep focused on other stuff. :D
14:04 JoeJulian Anyway... I suspect maybe amazon strips xfs support from the kernel so they don't supply the xfsprogs rpm.
14:05 JoeJulian That's just a guess though.
14:05 samsamm maybe i could just use mkfs.ext3?
14:05 JoeJulian Yep
14:05 JoeJulian or ext4
14:05 recidive joined #gluster
14:05 JoeJulian I would use ext4
14:05 JoeJulian semiosis uses amazon, but he does it with ubuntu
14:07 JoeJulian You could also try, "rpm -i http://mirror.centos.org/centos-6/6/updates/x86_​64/Packages/xfsprogs-3.1.1-10.el6_4.1.x86_64.rpm"
14:07 glusterbot <http://goo.gl/rt9CvI> (at mirror.centos.org)
14:08 semiosis JoeJulian: see pm re: datasets
14:08 JoeJulian odd... no pm...
14:08 bugs_ joined #gluster
14:13 semiosis i said see pm before i actually pm'ed
14:14 * semiosis is like udp
14:15 samsamm getting 'operation failed' when trying to create a volume. is there any way to get more info?
14:15 semiosis the glusterd log file, usually /var/log/glusterfs/etc-glusterfs-glusterd.....log
14:15 semiosis bbl, meetings :/
14:16 samsamm http://pastebin.com/38TbizzX any ideas on this/
14:16 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
14:16 samsamm ?
14:16 JoeJulian I should file a bug on that and even submit a patch (low hanging fruit). That "operation failed" should say, "Operation failed. See %s for details."
14:17 glusterbot http://goo.gl/UUuCq
14:17 JoeJulian What was your create command?
14:18 samsamm sudo gluster volume create webs replica 2 transport tcp ec2-54-246-196-253.eu-west-1.​compute.amazonaws.com:/export ec2-54-229-127-150.eu-west-1.​compute.amazonaws.com:/export
14:18 JoeJulian Check "gluster peer status" on both servers to ensure the hostnames look like that.
14:19 samsamm on one of the severs it says no peers present
14:19 samsamm should i probe on that one too then?
14:20 JoeJulian Also, in production, you'll want to use hostnames that won't change...
14:20 samsamm yeah will sort that later
14:20 JoeJulian One has peers, the other none? That's wierd.
14:21 samsamm one has 3 peers. the other has 1?
14:22 samsamm all say Disconnected
14:22 JoeJulian Do you know what you did to get it into that state?
14:23 samsamm ive run the probe command a few times
14:23 glusterbot joined #gluster
14:23 samsamm possibly with or without sudo
14:24 JoeJulian The quick and dirty way to clean that up (on all servers): service stop glusterd; rm -f /var/lib/glusterd/peers/*
14:24 JoeJulian Once that's done, start glusterd again and probe again.
14:25 samsamm ok, ill try that
14:25 JoeJulian Since amazon gives you a different IP each time you start your instance, it's a good idea to use their dns service to assign a hostname that doesn't change between boots to avoid that issue.
14:25 samsamm thanks
14:26 samsamm yeah those domains are from the elastic ip's. so they should remain static
14:26 JoeJulian ok
14:27 samsamm hmm. the probe seems to timeout
14:27 samsamm its not doing anything
14:27 JoeJulian iptables?
14:27 samsamm furrrk
14:28 samsamm is there a known port range to allow?
14:30 kkeithley @ports
14:30 glusterbot kkeithley: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
14:30 JoeJulian That needs updated for 3.4
14:32 JoeJulian @update ports 1 s/& up./& up for <3.4 and 49152 & up for 3.4./
14:33 JoeJulian @change ports 1 s/& up./& up for <3.4 and 49152 & up for 3.4./
14:33 glusterbot JoeJulian: Error: The command "change" is available in the Factoids, Herald, and Topic plugins. Please specify the plugin whose command you wish to call by using its name as a command before "change".
14:33 JoeJulian @factoids change ports 1 s/& up./& up for <3.4 and 49152 & up for 3.4./
14:33 glusterbot JoeJulian: The operation succeeded.
14:33 JoeJulian @ports
14:33 glusterbot JoeJulian: glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
14:34 JoeJulian @factoids change ports 1 s/(Deleted volumes do not reset this counter.) //
14:34 glusterbot JoeJulian: The operation succeeded.
14:37 jurrien_ joined #gluster
14:37 samsamm ive got it working! :)
14:37 samsamm im a little confused about the mount points though
14:38 samsamm how do i mount the share from an instance not in the gluster cluster?
14:38 JoeJulian mount -t glusterfs {any glusterfs server or rrdns entry}:{volume name} {mountpoint, ie. /mnt/foo}
14:39 samsamm but what happens if that one gluster server becomes unavailable?
14:39 samsamm how does the share remain up?
14:40 JoeJulian @mount server
14:40 glusterbot JoeJulian: The server specified is only used to retrieve the client volume definition. Once connected, the client connects to all the servers in the volume. See also @rrnds
14:40 samsamm ahh ok. makes sense
14:40 samsamm so each client will need gluster installed too?
14:41 zaitcev joined #gluster
14:42 JoeJulian Each client needs the glusterfs-fuse package installed.
14:42 samsamm ok
14:42 JoeJulian ... or you can mount via nfs, but you lose the redundancy the fuse client provides.
14:42 samsamm would nfs be faster?
14:49 jag3773 joined #gluster
14:55 daMaestro joined #gluster
14:57 JoeJulian yes and no
14:58 JoeJulian nfs caches directory listing and inode information through the kernel's FSCache. This allows faster lookup() times at the expense of inconsistency. It also has a slower throughput. It depends on your use case what's best for you.
15:02 manik joined #gluster
15:06 samsamm i have a 100mb file which im trying to open with vim. it just seems to crash. any ideas?
15:06 samsamm the file is in a gluster share
15:27 dbruhn joined #gluster
15:37 sprachgenerator joined #gluster
15:39 jebba joined #gluster
15:39 mooperd joined #gluster
15:41 mooperd joined #gluster
15:42 Guest11466 does anyone have problems on 3.4.0 with acl
15:42 Guest11466 I get different responses every time I run getfacl
15:42 Guest11466 I get 7796  17:39:36 getxattr("/kokot/users", "system.posix_acl_access", 0x7fff528829f0, 132) = -1 ENODATA (No data available)
15:43 chjohnst_work joined #gluster
15:43 Guest11466 8134  17:40:06 getxattr("/kokot/users", "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x07\x00\xff\xff​\xff\xff\x02\x00\x01\x00\x1a\x00\x00\x00\​x02\x00\x03\x000\x00\x00\x00\x02\x00\x03\​x00[\x00\x00\x00\x04\x00\x07\x00\xff\xff\​xff\xff\x10\x00\x07\x00\xff\xff\xff\xff \x00\x00\x00\xff\xff\xff\xff", 132) = 60
15:43 social any idea what could cause this?
15:44 JoeJulian Check the brick logs?
15:45 social I nothing at all :/
15:46 toad lol Guest94330 "kokot"
15:46 toad i mean social
15:48 toad and hi everyone / ahojte
15:48 JoeJulian <sigh> What would your grandmother say???
15:49 toad lets hope his grandma doesnt use his glusterfs
15:50 social pardon
15:50 social anyway
15:50 social when I set diagnostic log level on both bricks and clients to debug
15:50 social it looks like it got consistent
15:51 social I don't understand this :/
15:51 Gilbs1 joined #gluster
15:51 toad maybe some cache
15:52 social but where?
15:52 social fuse?
15:52 toad just a random thought, no idea
15:55 JoeJulian When you set the log level, that forced a reload of the vol configuration. Perhaps there's a clue there...
15:57 _pol joined #gluster
16:07 rcheleguini hello all, please, is it possible to export subdirs from a volume and mount them via glusterfs? like described at: http://www.gluster.org/pipermail/gl​uster-users/2011-April/030278.html
16:07 glusterbot <http://goo.gl/fXnfxS> (at www.gluster.org)
16:08 rcheleguini example: mount -t glusterfs gluster1:/volume1/subdir1 /mountpoint
16:10 JoeJulian nope
16:20 mohankumar joined #gluster
16:24 awheeler joined #gluster
16:25 Gilbs1 I'm getting tons of [client3_1-fops.c:1495:client3_1_inodelk_cbk] 0-tcstorage-volume-client-7: remote operation failed: No such file or directory, remote operation failed: No such file or directory. Path: <gfid:e9ea58c5-1b11-41a7-be26-5bb0cb21f9d7> (00000000-0000-0000-0000-000000000000) errors.  Is this related to the "DHT misses are expensive" article?  (http://joejulian.name/blog/​dht-misses-are-expensive/)
16:25 glusterbot <http://goo.gl/A3mCk> (at joejulian.name)
16:27 hagarth joined #gluster
16:34 Mo_ joined #gluster
16:35 JoeJulian Looks likely. I'd need more info to be sure, but that could cause that.
16:35 awheele__ joined #gluster
16:40 manik joined #gluster
16:42 Gilbs1 Ah, what info do you need and i'll round it up?
16:43 sprachgenerator joined #gluster
16:49 JoeJulian What led to that warning? Debug (or trace) logs surrounding it on both the client and the bricks (+- no more than 1 second should be plenty).
16:54 Gilbs1 Good question, let me dive into the logs more, it's been going on for quite some time.
16:56 nonsenso joined #gluster
16:58 nonsenso anyone around?
16:58 jthorne joined #gluster
17:08 thomasle_ joined #gluster
17:18 jag3773 joined #gluster
17:50 JoeJulian hello
17:50 glusterbot JoeJulian: Despite the fact that friendly greetings are nice, please ask your question. Carefully identify your problem in such a way that when a volunteer has a few minutes, they can offer you a potential solution. These are volunteers, so be patient. Answers may come in a few minutes, or may take hours. If you're still in the channel, someone will eventually offer an answer.
17:50 JoeJulian nonsenso: ^^^ That
18:04 Mo__ joined #gluster
18:10 edoceo Can gluster 3.2.5 do a reblance fix-layout while online ?
18:13 awheeler joined #gluster
18:15 nonsenso JoeJulian: thanks.  i think i figured out my question, actually.  :)
18:15 edoceo Also, I'm seeing some of this: 0-prod-replicate-0: background  entry self-heal failed on /image3/500321/500123/2013/04/
18:16 edoceo Does that mean my files are lost or ?
18:16 andreask joined #gluster
18:18 _pol joined #gluster
18:19 JoeJulian edoceo: probably not lost. That's a directory so as long as that directory exists on all your bricks, it's a spurious message. It can be fixed by resetting the trusted.afr ,,(extended attributes) for that directory directly on the bricks.
18:19 glusterbot edoceo: (#1) To read the extended attributes on the server: getfattr -m .  -d -e hex {filename}, or (#2) For more information on how GlusterFS uses extended attributes, see this article: http://goo.gl/Bf9Er
18:20 JoeJulian nonsenso: Sometimes figuring out your question is the part that leads you to the answer. :)
18:21 kkeithley JoeJulian, ninja philosopher
18:22 nonsenso JoeJulian: totally.
18:22 edoceo Thank you, what about this one: which is thrown when my GlusterFS gets really slow
18:22 edoceo [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xa0) [0x7faaced6d870] (-->/usr/lib/libgfrpc.so.0(rpc​_clnt_connection_cleanup+0x7d) [0x7faaced6d51d] (-->/usr/lib/libgfrpc.so.0​(saved_frames_destroy+0xe) [0x7faaced6d47e]))) 0-management: forced unwinding frame type(glusterd clnt mgmt) op(--(2)) called at 2013-08-09 11:04:23.16656
18:23 nonsenso tho, as conversation - i was at a conf and ran into a PM for gluster.  there was some mention about using replication factor of 3 isn't recommended.  is there validity to this?
18:23 JoeJulian I've been using replica 3 since 2.0.
18:23 edoceo It seems like my Gluster bricks are getting disconnected from each other?  I'm operating over GigE
18:23 JoeJulian but that's kind-of true. They didn't test more than replica 2 so they didn't want to guarantee anything.
18:24 nonsenso JoeJulian: i was surprised to hear that.  i haven't found anything in the docs or blogs to suggest why that is?
18:24 nonsenso got it.
18:24 JoeJulian And there have been bugs related to having more than two replicas, though every known one has been squashed.
18:24 aliguori joined #gluster
18:25 kkeithley I'm sure the PM was thinking RHS, where replica 3 is in "tech preview." IOW try it and see, but unsupported in RHS.   Sometimes you have to wack 'em upside the head to get them into upstream, community mode
18:27 kkeithley Unless perhaps you're actually using RHS.
18:27 JoeJulian Even hagarth and avati were saying that more than replica 2 was unadvised for a while.
18:28 kkeithley no doubt for good reason
18:28 JoeJulian Usually
18:28 JoeJulian But I'm a rebel.
18:28 edoceo Hm and after I see those messages, then operations on the mountpoint all hang
18:30 nonsenso kkeithley: what's RHS?
18:31 JoeJulian edoceo: 0-management, so that's not directly related to your fs hang. More like a symptom. I would guess that your brick is not responding. Maybe the brick filesystem? Maybe one of the other myriad of bugs that have been fixed since then?
18:31 JoeJulian 3.2 hasn't had a patch against it since Mar 19, 2012.
18:32 edoceo Is the upgrade to 3.3 series pretty painless?
18:32 JoeJulian yes
18:32 JoeJulian but it does require downtime.
18:32 edoceo Wer'e on ubuntu precise, just drop in new .deb packages?
18:32 edoceo I'd like to talk about down time, rather than have my FS act "funny" - that's an issue I can manage
18:33 JoeJulian I'm actually of the mind to recommend making the jump straight to 3.4... I've been very surprised at the lack of bugs and the quick adoption.
18:33 nonsenso JoeJulian: how long have you been running 3.4?
18:33 edoceo Oh golly, well - I suppose it couldn't be worse than what I'm getting now
18:34 edoceo I suppose I could drop 3.4 on there, and if it goes to crap I can rollback to my known 3.2.5 version?
18:34 JoeJulian I, personally, haven't had the time to even upgrade 3.3 recently, much less go to 3.4. My personal workstation running fedora 19 is using the 3.4 client though.
18:34 JoeJulian edoceo: yes.
18:35 edoceo Here we go!
18:35 JoeJulian edoceo: the only thing you'd need to do to roll back is to delete the .glusterfs tree from your bricks.
18:36 edoceo With a combination of `find ... -exec ` spell?
18:36 edoceo -exec rm that is
18:36 JoeJulian rm -rf $brickroot/.glusterfs
18:37 NuxRo JoeJulian: are you still using gluster as /home?
18:37 JoeJulian yep
18:38 NuxRo how's that working out for you, any gotchas?
18:38 JoeJulian none
18:38 NuxRo cool, might implement something similar at work
18:38 nonsenso /home on a laptop with replication or something?
18:39 NuxRo not really, more like a workstation client
18:39 NuxRo but you just gave me an idea now :)
18:39 JoeJulian /home for my linux workstations at the office.
18:41 NuxRo JoeJulian: question, if i created a raw image file on top of gluster and mounted it from several locations, would that work?
18:41 NuxRo i am not sure if locking would work in this situation
18:42 nonsenso NuxRo: someone mentioned they did this and had poor results.  i totally spaced on the details but if you try it - i'd love to hear how that works out for you.
18:42 JoeJulian Depends on what's on that image and how you're trying to use it. But if it's a typical writable filesystem, no.
18:42 NuxRo I thought of using it as a writable filesystem, yes, and luks-ified
18:42 NuxRo maybe ecryptfs on top of a regular volume would work better
18:43 JoeJulian Maybe with gfs2 on it it might work, but otherwise, no. The filesystem would have to expect inodes to change unexpectedly.
18:44 NuxRo right, had a hunch it wouldnt work but couldnt quite explain it, thanks
18:44 NuxRo nonsenso: so apparently it would be a nonsense to try it ;-)
18:44 JoeJulian You could lock, mount, modify, unmount, unlock, but then why not just use an object store...
18:44 NuxRo yep
18:46 edoceo The instructions for 3.4 reference following steps in 3.3 - does that include starting glusterd with upgrade=on ?  3.4 docs make no mention of that
18:47 chjohnst_work what is upgrade=on?
18:50 edoceo glusterd --xlator-option *.upgrade=on -N
18:50 JoeJulian chjohnst_work: That was something that had to be done to upgrade from 3.1 or 3.2 to 3.3. I imagine that would need to be done for the move to 3.4, too, but I haven't seen any documentation about that.
18:51 JoeJulian Worst case, the option no longer would exist. If it does, then it should do what it was designed to do.
18:51 chjohnst_work I already upgraded my 3.3 to 3.4 on one of my test nodes and I didnt do that, I am not sure its needed
18:51 JoeJulian Right, 3.3->3.4 doesn't need that.
18:51 chjohnst_work great!
18:51 JoeJulian But there were some structure changes from 3.2 to 3.3
18:51 chjohnst_work was going to start playing around with the quorum feature
18:52 jiku joined #gluster
18:52 awheele__ joined #gluster
18:56 edoceo Yea, I'm doing a 3.2.5 => 3.4 upgrade, so I did run that option, the output was "CMA: unable to get RDMA device list"
18:56 edoceo I'm guessing it went ok, the exit code was zero
18:57 edoceo But, I end up with two scripts in /etc/init.d/glusterfsd and /etc/init.d/glusterfs-server - do I need to start both?
18:58 JoeJulian No, the former is simply used to shutdown the bricks during init 0 and 6
19:05 edoceo Dang, well, it's lost all my volumes :(
19:06 edoceo Looks like stuff from /etc/glusterd/vols/ did not get copied/moved to /var/lib/glusterd/vols
19:06 JoeJulian Ah, right... I blame semiosis
19:06 edoceo Perhaps I'll just move the files manually and cross my fingers?
19:06 JoeJulian You can. Make sure glusterd is stopped.
19:06 semiosis ,,(3.3 upgrade notes)
19:06 glusterbot http://goo.gl/qOiO7
19:07 * semiosis absolved
19:07 JoeJulian Then do the upgrade=on again.
19:07 JoeJulian hehe
19:08 edoceo Hm, so I should not just move the vols/ stuff - I should move other things as well?
19:09 edoceo Seems it's missing some stuf in peers/ as well
19:09 JoeJulian mv /etc/glusterd /var/lib/glusterd
19:10 edoceo Winner!
19:11 JoeJulian You can blame me if you want. I keep making them adhere to FHS standards. :D
19:11 JoeJulian FSH
19:13 nightwalk joined #gluster
19:15 edoceo I'll blame both of you, so everyone wins :D
19:15 JoeJulian That's fine, but you still owe us each a beer.
19:15 edoceo Except me, `volume status prod` is hanging, 'prod' is my volume name
19:15 Gilbs1 Debug (or trace) logs surrounding it on both the client and the bricks (+- no more)  glusterd --debug, can i set a time limit?  I don't see it in the --usage.
19:16 JoeJulian No
19:16 JoeJulian edoceo: is glusterd running on your servers?
19:17 edoceo Yes, all of them have this: /usr/sbin/glusterd -p /var/run/glusterd.pid
19:19 edoceo netstat shows a bunch of stuff in the Send-Q to my other bricks, like the network isn't pushing my bits across
19:19 JoeJulian bad network...
19:20 JoeJulian I wonder if it's lagged with self-heal...
19:20 edoceo My log levesl are on Warning or Error only - safe to stop and turn that up to see more spew?
19:21 JoeJulian .... anything in the logs already?
19:21 JoeJulian I'm particularly thinking of glustershd.log
19:23 JoeJulian Maybe I should come down there and make it submit...
19:23 JoeJulian Are you running this at your place down there in Ballard?
19:24 edoceo Nope, it's in San Jose (CA, USA)
19:24 edoceo But you can come down anyways
19:24 edoceo I will get beer
19:25 JoeJulian I ride my bike all the way home on Wednesdays from Sodo so I'm only a few blocks away then...
19:25 * JoeJulian is in Edmonds.
19:26 edoceo yea, that's quite a ride - 20+ miles?
19:26 JoeJulian 23
19:27 edoceo Should I have a file called glustershd.log ?
19:27 Gilbs1 Great way to save on gas, it's a bit high out there in CA.
19:29 edoceo Hey, got something from /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
19:29 edoceo shows 0-management: connection attempt failed (Connection refused)
19:33 JoeJulian ,,(paste) /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
19:33 glusterbot For RPM based distros you can yum install fpaste, for debian and ubuntu it's pastebinit. Then you can easily pipe command output to [f] paste [binit] and it'll give you a URL.
19:35 JoeJulian Gilbs1: Nah, it's not to save on gas, it's to get this ass out of a chair and get some exercise to try to get my cholesterol back down to a manageable level.
19:36 JoeJulian ... and last summer when I was driving through Colorado, your prices weren't any better than they are here in the Seattle area. :D
19:37 edoceo Joe: http://pastebin.com/i8WCh3Wp
19:37 glusterbot Please use http://fpaste.org or http://dpaste.org . pb has too many ads. Say @paste in channel for info about paste utils.
19:37 edoceo Oh, sry
19:38 JoeJulian Heh
19:38 Gilbs1 JoeJulian: http://tny.cz/94417948   --  debug output from client/server
19:38 glusterbot Title: glusterd --debug - 94417948 (at tny.cz)
19:39 JoeJulian 203's the other server?
19:40 edoceo Yes, servers are on 201, 202 and 203
19:40 dhsmith joined #gluster
19:40 edoceo The log I sent was from 202
19:41 Gilbs1 Our gas prices are great, as long as you don't drive anywhere.  :)
19:42 JoeJulian edoceo: Let's see the one from 203 since that's the one that says it's refused.
19:42 JoeJulian Might want to double check you ,,(ports) are open, too.
19:42 glusterbot glusterd's management port is 24007/tcp and 24008/tcp if you use rdma. Bricks (glusterfsd) use 24009 & up for <3.4 and 49152 & up for 3.4. (Deleted volumes do not reset this counter.) Additionally it will listen on 38465-38467/tcp for nfs, also 38468 for NLM since 3.3.0. NFS also depends on rpcbind/portmap on port 111.
19:43 edoceo Which of these: http://edoceo.io/paste?p=5205465252108 - files do you want?
19:44 edoceo I've got no firewalls, and the system are connected to each other via unmanaged switch
19:44 JoeJulian Now there's the way to do it. :D
19:44 Gilbs1 lol
19:45 JoeJulian etc-glusterfs-glusterd.vol.log
19:46 JoeJulian Gilbs1: If you guys would stop sending all your renewable energy into the atmosphere... ;) We passed the sign that said, "Entering Colorado" and my son pipes up and says, "Who brought the marshmallows?"
19:46 edoceo Log: http://edoceo.io/paste?p=520546fc9681e
19:47 Gilbs1 Yep, my in-laws were close to losing their place and stayed with us for a week.  All is well and they go to go home. phew...
19:48 edoceo To the best of my knowledge I do not have any RDMA stuff
19:48 Gilbs1 This fall all the hunters are  in luck, the meat has been pre-smoked.
19:48 JoeJulian Nah, any RDMA messages can be safely ignored. If it can't find it, it turns it off.
19:49 JoeJulian reading from socket failed. Error (No data available), peer (192.168.100.201:24007)
19:49 myugetsudo Ah, 3.4 changed the port enumeration range?
19:49 myugetsudo Good to know.
19:49 JoeJulian they did...
19:49 edoceo 201 is another system in the set, it's started and listening on that port
19:50 JoeJulian edoceo: Double check all your "glusterd -V"
19:50 edoceo All report: glusterfs 3.4.0 built on Jul 12 2013 20:36:44
19:50 johnmwilliams joined #gluster
19:51 edoceo If I telnet to those ports (24007) it connects, but of course disconnects me when I type in "fdfdsafdsafsdkjlfds"
19:51 johnmwilliams I just setup Gluster/UFO for the first time on CentOS 6.4 and I am getting the following errors:
19:51 johnmwilliams https://gist.github.com/johnw​illiams/5a0119112b2d0f2fd647
19:51 glusterbot <http://goo.gl/kNBrHf> (at gist.github.com)
19:52 edoceo The netstat command is showing some bits in the SendQ side on all the machine, same number for at least 30s
19:53 edoceo Issue?  Extended attribute trusted.glusterfs.volume-id is absent
19:54 edoceo Initialization of volume 'prod-posix' failed, review your volfile again
19:57 JoeJulian edoceo: That would do it...
19:57 ricky-ticky joined #gluster
19:57 JoeJulian @bz search volume-id
19:58 edoceo Should I manually update the vol file for that thing?
19:58 JoeJulian No, it's an extended attribute
19:58 JoeJulian @bugzilla search volume-id
19:58 JoeJulian come on glusterbot...
20:00 * JoeJulian chuckles at himself for missing the obvious reference material: http://joejulian.name/blog/repl​acing-a-brick-on-glusterfs-340/
20:00 glusterbot <http://goo.gl/K7XS1G> (at joejulian.name)
20:02 edoceo Aha, so I just run that on each box, for each brick and I'll have a party!
20:02 JoeJulian +1
20:08 edoceo Oh, hot $hit!! I did that and `volume status prod` returns instantly, - would it make sense to try a heal and or re-balance at this poitn?
20:08 JoeJulian I would do a "heal $vol full"
20:13 sprachgenerator joined #gluster
20:14 johnmwilliams Is there a specific channel for gluster UFO?
20:14 JoeJulian Sorry, johnmwilliams, no. I've had a $dayjob fire that I'm putting out... Not trying to ignore you.
20:15 johnmwilliams JoeJulian: Thanks. Much appreciated.
20:22 edoceo There are some stuffs left-ver in /usr/lib/glusterfs/glusterfs/python/syncdaemon - after a remove,
20:26 edoceo And dang, now one of my hosts is showing no volumes
20:33 edoceo Ok, on `gluster peer status` shows no peers on C1 and C2 but on C3 it indicates that C1 and C2 are peers
20:34 JoeJulian Theoretically c3 should be able to probe c1 and c2 again and fix that.
20:35 JoeJulian Gah.. I'm gone. Probably for the day.
20:35 JoeJulian @meh
20:35 glusterbot JoeJulian: I'm not happy about it either
20:35 edoceo Hmm, I did that but it's still not showing them, volume sync?
20:36 JoeJulian Sounds like /etc/glusterd/peers didn't get moved.
20:36 johnmark johnmwilliams: you should check in #gluster-swift
20:36 JoeJulian johnmark: there's a #gluster-swift???
20:36 johnmark JoeJulian: yes :)
20:36 edoceo Oh, to /var/lib/gluster -
20:37 johnmark johnmwilliams: but I don't know if anyone is actually in there right now
20:38 Ramereth so i've searched the docs everywhere on this, but how do you set/use the user/pass auth on gluster volumes?
20:40 Ramereth i found one doc in the source repo but I couldn't figure out if you have any way to set it via the cli or not
20:41 dewey joined #gluster
20:44 * Ramereth waves at johnmark and JoeJulian
20:44 semiosis i've been wondering that too, Ramereth
20:45 semiosis but haven't yet looked into it
20:45 Ramereth is it not a stable feature?
21:04 Gilbs1 JoeJulian: http://tny.cz/94417948   --  debug output from client/server   (not sure if you seen this)
21:04 glusterbot Title: glusterd --debug - 94417948 (at tny.cz)
21:23 y4m4 Ramereth: not available through cli yet
21:24 y4m4 Ramereth: you can open a bug agains't it - i can work on getting it in
21:36 awheeler joined #gluster
21:45 awheele__ joined #gluster
21:47 jag3773 joined #gluster
21:52 Gilbs2 joined #gluster
21:52 edoceo Well, now that I'm on 3.4 the 'volume status` command just hangs, no errors in any of the logs
21:55 Ramereth y4m4: cool, i'll look into it. thanks!
21:59 edoceo What should the value of the 'listen-port' option be?  For some of my volume files it's a value like 24012, in others it's a '0'
22:00 y4m4 edoceo: "0" ? cannot be zero
22:01 edoceo Dang, could you look at yours and see if the listen-port is the same for all bricks across your set?
22:01 edoceo Like Volume:brick1 is using same listen port on all servers
22:05 edoceo My C1 machine has its two brick listening on 24011 and 24012, but C2 has it's bricks listening on same ports, so does C3
22:05 edoceo But the volume files on C1 say that only C1 bricks listen on those ports, the vol files from C2 bricks (that are in the /var/lib/glusterd/vols/ on C1) show listen-port=0
22:10 fidevo joined #gluster
22:13 y4m4 edoceo: a bug
22:13 y4m4 edoceo: which version are you using?
22:13 edoceo 3.4
22:13 edoceo Just got from the PPA
22:14 edoceo How can I fix this? Manuall configure the ports across my Bricks?
22:14 y4m4 edoceo: recreate the volume
22:14 y4m4 edoceo: and see if it fixes it
22:14 edoceo dur, that won't lose data right?  And also, I'm not sure how to drop and re-create a volume
22:16 tjstansell i've been having issues trying to rebuild a node and have it rejoin the cluster... and i'm seeing volume status show ports as N/A ... and the vol file has listen-port=0 too
22:16 tjstansell this is on 3.4.0, btw.
22:18 tjstansell on an entirely different cluster, my 02 host has listen-port=49153 for it's own brick and 0 for the brick on the 01 host ...
22:18 tjstansell and the 01 host has listen-port=0 for both bricks.
22:21 tjstansell on a 3rd cluster (2-node replica), the 01 host has listen-port=0 for it's own bricks and 4915x for the bricks on the 02 host and the 02 host has listen-port=0 for it's own bricks and 4915x for the bricks on the 01 host.
22:22 y4m4 edoceo: nope
22:22 y4m4 edoceo: unless you exactly re-create the topology
22:22 tjstansell so 3 separate, but identially configured, sites and they're in 3 different states.
22:22 y4m4 looks like a regression in 3.4.0
22:23 edoceo So, if I recreate the same way my data would be safe?
22:23 edoceo Do I just stop gluster on say, C3 and then wipe the /var/lib/glusterd and then re-probe C1 and volume sync ?
22:23 y4m4 edoceo: that would be easier
22:24 Ramereth sadface, auth.allow doesn't support cidr :(
22:24 edoceo How long should this volume sync take, instant right?
22:24 y4m4 edoceo: yeah
22:24 y4m4 edoceo: close to instant!
22:25 y4m4 Ramereth: as i said you 'bugzilla' can you your friend ;-)
22:25 y4m4 be*
22:25 Ramereth y4m4: indeed
22:25 edoceo Hm, is hanging....will try gain
22:25 y4m4 edoceo: do you see
22:26 y4m4 edoceo: listen-port as 0 on only on C3?
22:26 edoceo Nope that is showing up in some files on other machines as well.
22:27 y4m4 edoceo: in total you have c1, c2, c3 as servers.. or are there more?
22:27 edoceo c1,2 and 3
22:27 edoceo And the volume files on each server show liten-port only for their own bricks
22:27 edoceo Not for the others
22:28 edoceo Like this: http://edoceo.io/paste?p=52056cf84664a
22:30 y4m4 edoceo: weird
22:30 edoceo I think so too, desperate to get my system back online
22:31 y4m4 edoceo: i would ask you to re-create the volume
22:31 y4m4 edoceo: that way we can see if this is regular or transient issue
22:31 edoceo Ok, but I don't know the steps properly, guide me?
22:31 tjstansell i restarted glusterd on one of my nodes like yours that had listen-port=0 for all bricks except for the ones locally and after restarting they all show up as listen-port=0
22:32 y4m4 edoceo: 'gluster volume stop <volname>'
22:32 edoceo From like C1 then? or does that not matter
22:32 y4m4 edoceo: that doesn't matter
22:33 edoceo Ok, it's hanging....
22:33 y4m4 edoceo: ah then your 'glusterd' is down or not responding
22:33 edoceo I ran from C1 and I have glusterd running on C1 2 and 3
22:34 y4m4 edoceo: can you 'tail -10 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log" ?
22:34 y4m4 edoceo: on C1
22:35 edoceo http://edoceo.io/paste?p=52056e94a16ef
22:35 glusterbot Title: Pastebin (at edoceo.io)
22:35 y4m4 edoceo: /etc/init.d/glusterd on perhaps all and see if 'gluster peer status' everyone in trusted pool is listed
22:36 y4m4 edoceo: if yes then perhaps 'move ahead' with 'volume stop'
22:36 y4m4 edoceo: i meant restart 'glusterd'
22:36 edoceo Gluster Peer status show everyone is talking
22:37 y4m4 edoceo: okay then perhaps 'ps -ef | grep glusterfs' on c1, c2, c3.. pastebin?
22:37 edoceo When I do a stop, there are a lot of glusterfsd left over - hah -lag, going to paste
22:38 y4m4 edoceo: yeah they are because 'glusterd' is reading wrong configs
22:38 y4m4 edoceo: in /var/lib/glusterd' and has no way to killing it
22:38 edoceo http://edoceo.io/paste?p=52056f5d78ce1
22:38 glusterbot Title: Pastebin (at edoceo.io)
22:39 y4m4 edoceo: all have proper ports assigned but /var/lib/glusterd is inconsistent
22:40 edoceo Maybe I'll just rsync that /var/lib/glusterd from C1 to C2 and C3 - then it's time to start drinking
22:40 y4m4 edoceo: that won't work
22:40 edoceo Well, I'm sure the drinking will help
22:40 y4m4 edoceo: that too :P
22:41 y4m4 edoceo: you can manually edit the files which say listen-port=0
22:41 y4m4 edoceo: and specify proper ports
22:41 edoceo Ok, and I make them all match up right?
22:41 y4m4 edoceo: but i was hoping to get it right with 'glusterd'
22:42 edoceo I can try a few more Glusterd things if you'd like...I want to be helpful to the community, but I also want my FS back online
22:42 y4m4 edoceo: to get it right the only way is 'killall glusterfs;killall glusterfsd', 'gluster volume stop <volname> force'
22:43 y4m4 edoceo: check with 'gluster volume stop <volname> force'
22:43 y4m4 edoceo: then re-create it
22:43 harish joined #gluster
22:43 edoceo And the re-create will make sure my data is OK?
22:44 y4m4 edoceo: and there is one more step
22:45 edoceo I don't need to kill glusterd ?
22:45 y4m4 edoceo: if you kill 'glusterd' you won't be able to cleanly delete the volume
22:46 edoceo Ok, I've issue the stop force command from C1
22:46 edoceo It is hanging here, slow....
22:48 edoceo So the command returnd, but when I check again I see: volume stop: stage: failed: Another transaction is in progress. Please try again after sometime
22:48 edoceo The volume is 'stage'
22:48 y4m4 edoceo: ah previous stop never completed
22:48 y4m4 successfully
22:48 y4m4 and 'glusterd' is waiting for it to complete
22:49 y4m4 it won't complete now since your /var/lib/glusterd is inconsistent
22:49 edoceo Shoot!  So, maybe it's a stop-everything time and then manually sync the /var/lib/glusterd stuff ?
22:50 harish joined #gluster
22:53 tjstansell fyi, if i stop all gluster processes on one node and restart glusterd which then restarts all the bricks, the listen-port gets updated to the correct values for those bricks.
23:04 mooperd_ joined #gluster
23:25 badone__ joined #gluster

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary