Camelia, the Perl 6 bug

IRC log for #gluster, 2013-06-22

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 fxmulder joined #gluster
00:08 jclift_ Ugh.  Need second pair of eyes.
00:08 jclift_ Even basic volume creation isn't working for me with the 3.4.0 beta 3 rpms.
00:09 jclift_ # gluster volume create test1 transport rdma gluster1-1:/export/brick2/glusterfs/
00:09 jclift_ volume create: test1: failed
00:10 jclift_ But /var/log/glusterfs/cli.log says: [2013-06-22 00:07:02.780261] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"
00:10 jclift_ I think I'll try compiling my own rpms from latest release-3.4 branch instead
00:11 hlieberman jclift_, We see those errors too.
00:11 hlieberman They're meaningless, I think.
00:12 jclift_ For some reason I can't even create basic volumes, regardless of transport type
00:12 jclift_ Well, I think these rpms are full of fail ;)
00:13 y4m4 joined #gluster
00:26 JoeJulian jclift_: could you fpaste the log?
00:26 jclift_ JoeJulian: Which log do you want?  I'm trying to find any useful info output by anything at all.
00:26 jclift_ JoeJulian: There's the cli.log, but that's about it.
00:26 JoeJulian probably client log
00:27 JoeJulian is rpc-transport.c even used in the cli?
00:27 JoeJulian Oh, it is...
00:27 JoeJulian ok...
00:28 JoeJulian So that warning you posted was benign.
00:28 jclift_ Cool.
00:28 jclift_ This is the entire cli.log (not much): http://fpaste.org/20269/
00:28 glusterbot Title: #20269 Fedora Project Pastebin (at fpaste.org)
00:29 jclift_ I nuked the /var/log/glusterfs/ dir before doing it again this time, just so there's clean logs
00:29 JoeJulian cool
00:30 jclift_ Note, this also happens without the "transport rdma" thing, so it's not specific to the rdma code
00:30 JoeJulian So far so good. It's accepting the probe and peer creation.
00:31 JoeJulian Does peer status from both servers look good?
00:31 jclift_ Any idea how to increase the verbosity from the gluster cli, so it gives verbose or debug level output?  There's nothing in the man page
00:31 jclift_ Yeah, peer status seems fine
00:32 jclift_ Both of the gluster nodes have dual port IB cards, both ports running IPoIB
00:32 jclift_ They've also got single ethernet nics too.
00:32 JoeJulian You can use the --log-level= flag
00:33 jclift_ k, I'll try that.  Suggestion for good log-level for debugging this kind of thing?
00:44 vpshastry joined #gluster
00:46 jclift_ JoeJulian: This is the cli.log with DEBUG level log: http://fpaste.org/20270/13718619/
00:46 glusterbot Title: #20270 Fedora Project Pastebin (at fpaste.org)
00:47 jclift_ JoeJulian: Not seeing any indication of why "failure" is happening :(
00:49 JoeJulian yeah, that doesn't really tell us anything.
00:49 JoeJulian How about both glusterd.vol.log
00:50 jclift_ http://fpaste.org/20272/37186223/
00:50 glusterbot Title: #20272 Fedora Project Pastebin (at fpaste.org)
00:51 jclift_ There seems to be a severe lack of info in the logs here. :(
00:51 JoeJulian That's all that's in either one?
00:52 jclift_ Yep
00:52 jclift_ "effectively useless" :(
00:52 JoeJulian That doesn't make any sense...
00:52 jclift_ I fully agree
00:53 jclift_ I'm writing up a BZ about it now.  Maybe someone's already seen it.
00:53 JoeJulian It shows the callback so that should mean that glusterd responded, I would think.
00:53 jclift_ I kind of suspect that the code is getting an "it's all ok" signal, and somehow mistakenly thinking "whoa, problem here, abort".
00:53 jclift_ Anyway, whoever deals with the BZ can look into it. ;)
00:54 jclift_ I can give them remote log in via ssh if they need it, etc.
00:54 JoeJulian If you use the --remote-host= flag and talk to the other server, you could at least look in wireshark to be sure.
00:54 JoeJulian Is that an internal box?
00:55 ShaharL joined #gluster
00:55 jclift_ Yeah, this is all my personal dev/test lab gear that's next to my desk here.
00:55 jclift_ It's all plugged into 1 older model (but decent) DDR IB switch, and a cheapo TP-LINK 24 port eth switch.
00:56 JoeJulian I should have had johnmark send those 1Us to my house for testing... :D
00:56 jclift_ :D
00:56 * JoeJulian is only half joking...
00:57 * jclift_ bought the switch on eBay for £180
00:57 jclift_ Shipping was another £14.95 pounds
00:57 JoeJulian I'm just woefully short on cpu and memory right now...
00:57 jclift_ Personally, I like this switch because it can run fanless.  So, total silence. :)
00:58 JoeJulian nice
00:58 jclift_ Yeah.  The boxes themselves are just self assembled consumer grade mini towers.
00:58 JoeJulian But I don't think rdma has anything to do with the problem you're currently having...
00:58 JoeJulian selinux?
00:58 jclift_ Interesting thought.  Maybe.
01:01 Shahar joined #gluster
01:01 cakes joined #gluster
01:04 jclift_ SELinux doesn't show anything in the audit log.
01:04 jclift_ Hmmm, I'll disable it and retry.
01:06 bulde joined #gluster
01:07 ShaharL joined #gluster
01:21 jclift_ No difference at all.
01:21 jclift_ Well, it's not SELinux
01:21 JoeJulian hrm
01:21 JoeJulian You can run glusterd in the foreground/debug mode with -d
01:23 jclift_ Ahhh yeah, good point.
01:26 semiosis i'd like to file a bug, can you help me glusterbot
01:26 glusterbot http://goo.gl/UUuCq
01:26 semiosis thanks
01:26 jclift_ JoeJulian: http://fpaste.org/20274/37186437/
01:26 glusterbot Title: #20274 Fedora Project Pastebin (at fpaste.org)
01:27 jclift_ JoeJulian: That's from glusterd in --debug mode
01:27 JoeJulian jclift_: Was this trying to create the volume, "test2"?
01:27 jclift_ Yep
01:28 jclift_ $ sudo gluster volume create test2 transport rdma gluster1-1:/export/brick2/glusterfs --log-level=DEBUG
01:28 semiosis JoeJulian: fyi, https://launchpadlibrarian.net/143078766/bu​ildlog_ubuntu-precise-amd64.glusterfs_3.4.0​beta3-beta3~precise3_FAILEDTOBUILD.txt.gz
01:28 glusterbot <http://goo.gl/XZwTd> (at launchpadlibrarian.net)
01:28 jclift_ This doesn't happen with git master
01:28 JoeJulian jclift_: which part of /var/lib/glusterd/vols/test2 doesn't exist?
01:28 semiosis BD xlator breaks build on debian, filing a bug now
01:28 JoeJulian oops
01:29 jclift_ JoeJulian: /var/lib/glusterd/vols/ is empty.
01:29 jclift_ There are no other volumes on the box
01:29 * jclift_ clears them out each time
01:30 jclift_ JoeJulian: Meh, I think I've about had it with this.  I'll just dump all of this info into the BZ, and let someone else care about it. :)
01:30 JoeJulian Ok, guess that's not odd...
01:30 JoeJulian ok
01:30 JoeJulian Oh!
01:30 jclift_ JoeJulian: Thanks for the assistance.
01:30 jclift_ Oh?
01:30 JoeJulian Unable to find friend: gluster1-1
01:30 JoeJulian Did you probe back to the first server to assign its hostname?
01:31 jclift_ Nah, the ip address it lists right after that is the correct IP for gluster1-1
01:31 jclift_ Nope.
01:31 JoeJulian That's the problem.
01:31 JoeJulian @hostname
01:31 glusterbot JoeJulian: I do not know about 'hostname', but I do know about these similar topics: 'hostnames'
01:31 JoeJulian @hostnames
01:31 glusterbot JoeJulian: Hostnames can be used instead of IPs for server (peer) addresses. To update an existing peer's address from IP to hostname, just probe it by name from any other peer. When creating a new pool, probe all other servers by name from the first, then probe the first by name from just one of the others.
01:31 jclift_ I'm *doing* this from the gluster1 box
01:31 JoeJulian The last phrase is important.
01:32 JoeJulian ... and I hate it.
01:32 jclift_ Why does 3.4.0 beta 3 need me to probe the box I'm adding the volume on, when no other build I've ever done needs it?
01:33 JoeJulian you didn't use hostnames before?
01:33 semiosis JoeJulian: do you know if the component "BDB" in bugzilla refers to the BD xlator?
01:33 JoeJulian semiosis: nope. That's an old xlator
01:33 semiosis ok thanks
01:33 jclift_ JoeJulian: I'm using the exact same commands as previous gluster installs.  Used them right out of command history.
01:33 jclift_ Meh, I'll try it anyway
01:33 JoeJulian jclift_: Then you've been doing it wrong. ;P
01:33 stopbit joined #gluster
01:33 jclift_ "doing it wrong" has always worked so far. :)
01:34 jclift_ But, ok
01:35 jclift_ You know, I've often thought that Gluster's understanding of hostnames vs IP's is seriously braindamaged/broken in some fundamental way.
01:35 forest joined #gluster
01:35 JoeJulian +1
01:35 jclift_ I really wonder wtf someone hasn't fixed it already.  Must be more complicated than it seems for some reason.
01:36 jclift_ Then again, I think the same thoughts about volume creation failure actually writing metadata to the mount point before success happens too... :D
01:41 jclift_ JoeJulian: Hmmm... so on the first host (gluster1-1) when I do peer status it's showing the hostname of the other box (gluster2-2).
01:42 jclift_ JoeJulian: On the 2nd host (gluster2-2), if I do a peer status it's showing the IP address for gluster1 (172.16.102.1).
01:42 JoeJulian It probably is a little bit. Should be easier once we have some sort of paxos cluster configuration management.
01:42 jclift_ JoeJulian: My reading of the @hostnames thing says that I can convert that IP address to a hostname by peer probing gluster1 by its name.
01:43 JoeJulian jclift_: That's precisely why you have to probe the first peer by name from any other peer.
01:43 jclift_ JoeJulian: k.  Should I see a change in behaviour from peer status then?
01:43 JoeJulian That's also why volume creation fails.
01:43 jclift_ k
01:43 jclift_ Talk about learning the hard way. :D
01:43 JoeJulian hehe
01:44 jclift_ Shouldn't this now be showing a hostname instead: http://fpaste.org/20275/86548013/  ?
01:45 glusterbot Title: #20275 Fedora Project Pastebin (at fpaste.org)
01:45 JoeJulian yep... <sigh>
01:45 JoeJulian That's a bug. Try restarting both glusterd?
01:45 jclift_ I'll reboot both boxes for good measure.
01:55 jclift_ JoeJulian: Doh, not a bad.  Idiot me.
01:56 jclift_ I'd was using the IP of one port on the host, but giving the hostname of the other port of the host.
01:56 jclift_ All of today has been this kind of day.  I knew I needed more sleep today. :/
01:57 jclift_ s/not a bad/not a bug/
01:57 glusterbot What jclift_ meant to say was: JoeJulian: Doh, not a bug.  Idiot me.
01:57 jclift_ My typing is getting really bad.
01:59 JoeJulian Heh, you better go sleep. :D
01:59 jclift_ Yeah.  I'm just verifying that this fixes it before I hit the sack.
02:02 glusterbot New news from newglusterbugs: [Bug 976946] BD xlator causes build to fail on debian due to implicit int to pointer conversion <http://goo.gl/8tAin>
02:02 jclift_ Success
02:03 jclift_ Now I'm wondering if I should file a bug about us needing more informative error text for this kind of thing. ie where the gluster node on the mount create command doesn't correspond to a known peer
02:03 glusterbot http://goo.gl/UUuCq
02:04 jclift_ Silent-ish failure is kind of crappy
02:04 JoeJulian yes!
02:18 Shahar joined #gluster
02:18 jclift_ k, wrote up a BZ for it.  Should be reasonably descriptive. https://bugzilla.redhat.com/show_bug.cgi?id=976948
02:18 glusterbot <http://goo.gl/UWCSq> (at bugzilla.redhat.com)
02:18 glusterbot Bug 976948: low, unspecified, ---, kaushal, NEW , Complete lack of informative information when volume creation fails due to using wrong IP of a known gluster host
02:19 jclift_ Hopefully someone makes a fix for this, so future people get told when they're doing it wrong :)
02:27 Shahar joined #gluster
02:29 ShaharL joined #gluster
02:32 glusterbot New news from newglusterbugs: [Bug 976948] Complete lack of informative information when volume creation fails due to using wrong IP of a known gluster host <http://goo.gl/UWCSq>
02:33 thisisdave joined #gluster
02:58 Shahar joined #gluster
03:03 ShaharL joined #gluster
03:12 vpshastry joined #gluster
03:15 vpshastry left #gluster
03:22 shanks` joined #gluster
03:26 hchiramm__ joined #gluster
03:27 shanks joined #gluster
03:32 hchiramm__ joined #gluster
04:01 RangerRick4 joined #gluster
04:02 ccha2 joined #gluster
04:46 hagarth joined #gluster
04:50 vpshastry joined #gluster
04:50 vpshastry left #gluster
06:37 nightwalk joined #gluster
06:38 ccha2 joined #gluster
06:49 ctria joined #gluster
07:02 hchiramm_ joined #gluster
07:02 shanks joined #gluster
07:02 CheRi joined #gluster
07:06 hchiramm__ joined #gluster
07:08 shanks` joined #gluster
07:08 CheRi_ joined #gluster
07:11 shanks joined #gluster
07:11 CheRi__ joined #gluster
07:12 hchiramm_ joined #gluster
07:35 forest joined #gluster
08:18 hagarth joined #gluster
08:53 forest joined #gluster
09:32 rcoup joined #gluster
09:53 mooperd joined #gluster
10:02 mooperd joined #gluster
10:04 andreask joined #gluster
10:41 isomorphic left #gluster
11:05 kevein joined #gluster
11:17 shanks joined #gluster
11:17 hchiramm_ joined #gluster
11:17 CheRi__ joined #gluster
11:32 hchiramm_ joined #gluster
11:32 shanks joined #gluster
11:32 CheRi__ joined #gluster
11:45 shanks joined #gluster
11:45 hchiramm_ joined #gluster
11:46 CheRi__ joined #gluster
12:29 yinyin joined #gluster
12:59 mohankumar joined #gluster
13:08 mohankumar joined #gluster
13:22 piotrektt joined #gluster
13:34 yinyin joined #gluster
13:39 forest joined #gluster
13:41 piotrektt joined #gluster
13:54 vpshastry joined #gluster
13:54 mohankumar joined #gluster
13:55 duerF joined #gluster
13:58 joelwallis joined #gluster
14:11 mohankumar joined #gluster
14:14 satheesh joined #gluster
15:12 _ilbot joined #gluster
15:12 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
15:44 _ilbot joined #gluster
15:44 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
16:24 yinyin joined #gluster
16:30 vimal joined #gluster
17:15 _ilbot joined #gluster
17:15 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
17:24 _ilbot joined #gluster
17:24 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
17:32 _ilbot joined #gluster
17:32 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
17:33 nightwalk joined #gluster
17:48 _ilbot joined #gluster
17:48 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
18:24 plarsen joined #gluster
18:31 JoeJulian vdrmrt_: That must be in 3.4 beta 3?
18:48 _ilbot joined #gluster
18:48 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/
20:46 _ilbot joined #gluster
20:46 Topic for #gluster is now  Gluster Community - http://gluster.org | Q&A - http://community.gluster.org/ | Patches - http://review.gluster.org/ | Developers go to #gluster-dev | Channel Logs - http://irclog.perlgeek.de/gluster/

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary