Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster, 2016-04-25

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:03 shyam joined #gluster
00:34 DV joined #gluster
00:38 devilspgd joined #gluster
00:43 om joined #gluster
00:50 devilspgd joined #gluster
01:06 russoisraeli joined #gluster
01:15 harish joined #gluster
01:30 EinstCrazy joined #gluster
01:33 harish joined #gluster
01:49 harish joined #gluster
02:01 harish joined #gluster
02:03 EinstCrazy joined #gluster
02:05 DV__ joined #gluster
02:11 cholcombe joined #gluster
02:12 harish joined #gluster
02:18 Lee1092 joined #gluster
02:20 harish joined #gluster
02:28 russoisraeli joined #gluster
02:38 julim joined #gluster
02:43 EinstCra_ joined #gluster
02:55 harish joined #gluster
02:56 julim joined #gluster
03:00 lynus joined #gluster
03:04 ramteid joined #gluster
03:05 harish joined #gluster
03:07 harish joined #gluster
03:17 harish joined #gluster
03:22 harish joined #gluster
03:24 harish joined #gluster
03:29 harish joined #gluster
03:36 shubhendu joined #gluster
03:41 mowntan joined #gluster
03:41 mowntan joined #gluster
03:42 itisravi joined #gluster
03:58 nbalacha joined #gluster
04:08 hchiramm joined #gluster
04:15 poornimag joined #gluster
04:19 kdhananjay joined #gluster
04:21 RameshN joined #gluster
04:22 skoduri joined #gluster
04:23 julim joined #gluster
04:29 julim joined #gluster
04:53 aspandey joined #gluster
04:56 rafi joined #gluster
05:01 karthik___ joined #gluster
05:05 ndarshan joined #gluster
05:10 overclk joined #gluster
05:10 hgowtham joined #gluster
05:10 gowtham joined #gluster
05:11 ashiq joined #gluster
05:18 nehar joined #gluster
05:21 hchiramm joined #gluster
05:21 ppai joined #gluster
05:24 spalai joined #gluster
05:26 jiffin joined #gluster
05:34 Bhaskarakiran joined #gluster
05:45 PaulCuzner joined #gluster
05:46 mhulsman joined #gluster
05:47 Apeksha joined #gluster
05:52 PaulCuzner joined #gluster
05:53 aravindavk joined #gluster
05:53 nishanth joined #gluster
05:56 karnan joined #gluster
05:59 harish_ joined #gluster
06:02 PaulCuzner joined #gluster
06:07 Manikandan joined #gluster
06:07 prasanth joined #gluster
06:10 Manikandan joined #gluster
06:15 ashiq joined #gluster
06:17 hchiramm joined #gluster
06:19 jtux joined #gluster
06:19 ppai joined #gluster
06:25 ramky joined #gluster
06:26 rastar joined #gluster
06:31 [Enrico] joined #gluster
06:33 kovshenin joined #gluster
06:37 kshlm joined #gluster
06:41 aravindavk joined #gluster
06:42 xMopxShell In a distributed volume, can gluster block read/writes when one or more bricks is offline?
06:43 anil_ joined #gluster
06:45 robb_nl joined #gluster
06:46 Pupeno joined #gluster
06:49 R0ok_ joined #gluster
06:50 Bhaskarakiran joined #gluster
06:51 jiffin xMopxShell: In a plain distribute volume , if bricks are offline, I/O's to that brick will be blocked
06:51 xMopxShell jiffin: what i'm experiencing is the files simply disappear
06:53 xMopxShell which is.... not good.
06:53 Saravanakmr joined #gluster
06:54 vmallika joined #gluster
06:54 jiffin xMopxShell: IMO , the files belong to offline brick will disappear.
06:55 jiffin xMopxShell: if u want consistency , u can try replicated, distributed replicated or disperse volume
06:56 xMopxShell jiffin: is it possible to block the entire volume if bricks are missing?
06:56 hackman joined #gluster
07:00 jiffin xMopxShell: I don't think so. What will be the benefit by doing so?
07:00 [Enrico] joined #gluster
07:01 xMopxShell jiffin: use cases where the presence/absence of a file triggers potentially destructive operations
07:03 mbukatov joined #gluster
07:06 jiffin xMopxShell: IMO use replicated or disperse volumes so that u can avoid those scenarios. You can get more wider opinions send out a mail gluster ML  by explaining our use case.
07:11 jiffin s/our/your
07:18 mhulsman joined #gluster
07:18 armyriad joined #gluster
07:19 jri joined #gluster
07:20 fsimonce joined #gluster
07:21 skoduri joined #gluster
07:23 DV joined #gluster
07:28 ctria joined #gluster
07:33 nbalacha joined #gluster
07:40 level7_ joined #gluster
07:45 mhulsman1 joined #gluster
07:51 aravindavk joined #gluster
07:55 EinstCrazy joined #gluster
07:55 armyriad joined #gluster
07:58 R0ok_ joined #gluster
07:59 rastar joined #gluster
08:03 JesperA joined #gluster
08:19 harish_ joined #gluster
08:35 itisravi joined #gluster
08:35 aspandey_ joined #gluster
08:36 robb_nl joined #gluster
08:37 kdhananjay joined #gluster
08:40 kassav joined #gluster
08:42 ashiq joined #gluster
08:46 Bhaskarakiran joined #gluster
08:54 Pupeno joined #gluster
08:57 Slashman joined #gluster
08:58 kovshenin joined #gluster
08:58 TvL2386 joined #gluster
09:00 Pupeno_ joined #gluster
09:04 paul98_ nn
09:07 kdhananjay joined #gluster
09:08 aspandey joined #gluster
09:12 nbalacha joined #gluster
09:28 level7 joined #gluster
09:31 Pupeno joined #gluster
09:44 skoduri joined #gluster
09:46 Pupeno joined #gluster
09:49 harish_ joined #gluster
09:54 arcolife joined #gluster
10:06 Pupeno joined #gluster
10:10 Slashman joined #gluster
10:18 robb_nl joined #gluster
10:24 atinm joined #gluster
10:24 level7_ joined #gluster
10:35 Debloper joined #gluster
10:40 post-factum why one could get "failed while binding to less than 65535 (Permission denied)" error with samba+glusterfs-vfs module?
10:42 ira joined #gluster
10:43 post-factum oh, selinux :/
10:44 karnan joined #gluster
10:59 level7 joined #gluster
11:09 nbalacha joined #gluster
11:16 russoisraeli joined #gluster
11:17 johnmilton joined #gluster
11:20 karnan joined #gluster
11:20 mhulsman joined #gluster
11:22 alghost_ left #gluster
11:32 ppai_ joined #gluster
11:32 kshlm joined #gluster
11:42 ppai_ joined #gluster
11:47 atinm joined #gluster
12:17 _ndevos joined #gluster
12:17 _ndevos joined #gluster
12:17 russoisraeli joined #gluster
12:18 julim joined #gluster
12:20 mhulsman1 joined #gluster
12:21 vmallika joined #gluster
12:25 ndevos joined #gluster
12:25 ndevos joined #gluster
12:29 pur__ joined #gluster
12:36 level7 joined #gluster
12:38 overclk joined #gluster
12:38 unclemarc joined #gluster
12:39 mhulsman joined #gluster
12:42 hackman joined #gluster
12:44 Guest50114 joined #gluster
12:48 plarsen joined #gluster
13:00 shaunm joined #gluster
13:02 nbalacha joined #gluster
13:02 Kins joined #gluster
13:03 azilian joined #gluster
13:04 tom[] joined #gluster
13:04 fale joined #gluster
13:04 mmckeen joined #gluster
13:05 post-factum ok, going to switch production samba from fuse mountpoint to gfapi
13:05 post-factum good luck to me :)
13:09 jbrooks joined #gluster
13:09 russoisraeli joined #gluster
13:11 ndevos post-factum: ai, it would be a little late to present your experiences at https://sambaxp.org then... maybe next year or an other conference?
13:13 post-factum ndevos: definitely not this year
13:13 post-factum ndevos: but i believe i should have a possibility to attend something starting from this autumn
13:14 ndevos post-factum: cool, we may meet at LinuxCon in Berlin then :)
13:14 ndevos http://events.linuxfoundatio​n.org/events/linuxcon-europe - beginning of october
13:15 glusterbot Title: LinuxCon Europe 2016 | Linux Conferences and Linux Events | The Linux Foundation (at events.linuxfoundation.org)
13:15 misc joined #gluster
13:16 post-factum quite an expensive event
13:17 bwerthmann joined #gluster
13:20 TvL2386 joined #gluster
13:20 mhulsman1 joined #gluster
13:22 mhulsman joined #gluster
13:22 Twistedgrim joined #gluster
13:25 ndevos post-factum: speakers enter for free, you just have to propose a talk, and if it is about Gluster, we (well, amye) might be able to sponsor travelcosts/hotel
13:25 ndevos of course, that ^ counts for others as well
13:28 post-factum ok, will consider that, thanks
13:30 misc also, you can do the ninja way, just get a tuxedo and say you are here to replace "robert" and pretend to be staff
13:31 lpabon joined #gluster
13:33 robb_nl joined #gluster
13:34 jiffin joined #gluster
13:36 rafi joined #gluster
13:40 mpietersen joined #gluster
14:01 post-factum :D
14:02 ndarshan joined #gluster
14:05 nbalacha joined #gluster
14:17 bennyturns joined #gluster
14:24 EinstCrazy joined #gluster
14:31 kshlm joined #gluster
14:37 post-factum joined #gluster
14:39 julim joined #gluster
14:42 post-factum joined #gluster
14:44 EinstCrazy joined #gluster
14:47 skylar joined #gluster
14:53 post-factum joined #gluster
14:58 wushudoin joined #gluster
15:01 wushudoin joined #gluster
15:01 shubhendu joined #gluster
15:07 sakshi joined #gluster
15:09 shaunm joined #gluster
15:13 nathwill joined #gluster
15:15 Slashman joined #gluster
15:22 uebera|| Hi. Can someone point me to an up-to-date example (using v3.7.11) of how to use SSL certificates?
15:22 uebera|| I have set up the following:
15:22 uebera|| - /var/lib/glusterd/secure-access, /etc/ssl/glusterfs.{ca,pem,key}  (the first contains all but the own server cert)
15:22 uebera|| - client.ssl on, server.ssl on, auth.ssl-allow 'SERVER02,SERVER04,SERVER05', ssl.certificate-depth [0|1|2]  (tried all of the latter to no avail)
15:22 uebera|| What I see in the logs is '0-webroot-client-1: SSL connect error', '0-webroot-client-1: client setup failed', 'peer CN = SERVER0n' (see the above list)
15:22 uebera|| The certificates work flawlessly with stunnel (mutual auth)...
15:27 robb_nl joined #gluster
15:31 paul98 joined #gluster
15:35 uebera|| (the above is for port 24007)
15:39 icholy joined #gluster
15:39 icholy is it a good idea to run postgres on gluster?
15:44 uebera|| icholy: I'm not using postgres, but maybe the following points you in the right direction --> https://www.gluster.org/pipermail/g​luster-users/2014-July/017934.html (this is not necessarily the last discussion on that topic on the list)
15:44 glusterbot Title: [Gluster-users] postgresql HA (at www.gluster.org)
15:49 kpease joined #gluster
15:55 DV joined #gluster
16:03 JoeJulian uebera||: Most up to date documentation lives at http://gluster.readthedocs.org/en/​latest/Administrator%20Guide/SSL/
16:03 glusterbot Title: SSL - Gluster Docs (at gluster.readthedocs.org)
16:04 Peppard joined #gluster
16:06 Guest3901 does gluster support reaching a brick via multiple IP addresses ? or is NIC bonding the only option? i google around and didn't have much luck finding an answer
16:10 JoeJulian If you use hostnames, it can resolve to whichever ip address you wish it to use, or it can use rrdns.
16:10 kpease joined #gluster
16:11 Pupeno joined #gluster
16:11 Pupeno joined #gluster
16:14 bluenemo joined #gluster
16:14 JoeJulian uebera||: Can you pastebin the snippit of log that has the ssl failure. I'm perusing the source and it looks like there's an error log line missing.
16:19 icholy left #gluster
16:29 rafi joined #gluster
16:38 DV joined #gluster
16:38 uebera|| JoeJulian: https://gluster.readthedocs.org/en​/latest/Administrator%20Guide/SSL/ still mentions "ssl.cert-depth" (which became "ssl.certificate-depth") and the "ssl.cipher-list 'HIGH:!SSLv2'" example does not mention the discussion about minimal best-practice settings on [Gluster-users] from a year ago (https://www.gluster.org/pipermail/gl​uster-users/2015-March/021139.html), that's why I'm asking.
16:38 glusterbot Title: SSL - Gluster Docs (at gluster.readthedocs.org)
16:38 Manikandan joined #gluster
16:39 JoeJulian <sigh>
16:42 JoeJulian uebera||: Could you file an issue for that please? https://github.com/gluster/glusterdocs/issues
16:42 glusterbot Title: Issues · gluster/glusterdocs · GitHub (at github.com)
16:43 EinstCrazy joined #gluster
16:47 uebera|| Will do.
16:51 jiffin joined #gluster
16:53 F2Knight joined #gluster
16:55 shubhendu joined #gluster
17:08 uebera|| Interesting. I *also* see the following in /var/lib/glusterd/*.vol (may have been a typo at one point, but how come "gluster volume info" does not show these? There I only see "ssl.certificate-depth: 1") :
17:08 uebera|| option transport.socket.ssl-cetificate-depth 1
17:09 uebera|| The logs complain about "W [graph.c:357:_log_if_unknown_option] 0-webroot-client-0: option 'transport.socket.ssl-cetificate-depth' is not recognized"
17:09 uebera|| (note the missing 'r' in 'certificate')
17:10 uebera|| On the CLI, I've only modified ssl.* options...
17:12 mhulsman joined #gluster
17:15 hchiramm joined #gluster
17:19 skylar joined #gluster
17:20 uebera|| JoeJulian: Issue https://github.com/gluster/glusterdocs/issues/105 filed.
17:20 glusterbot Title: Section 'Setting up GlusterFS with SSL/TLS' of the Administrator Guide is outdated · Issue #105 · gluster/glusterdocs · GitHub (at github.com)
17:21 bwerthma1n joined #gluster
17:24 Pupeno joined #gluster
17:29 JesperA joined #gluster
17:29 post-factum joined #gluster
17:30 frakt joined #gluster
17:37 uebera|| Regarding the above warning/error--that's a typo in the v3.7.11 sources (maybe applies to other versions as well):
17:37 glusterbot uebera||: warning/error's karma is now -1
17:37 uebera|| xlators/mgmt/glusterd/src/glusterd-volgen.c:2120:        RPC_SET_OPT(xl, SSL_CERT_DEPTH_OPT, "ssl-cetificate-depth", return -1);
17:37 uebera|| xlators/mgmt/glusterd/src/glusterd-volgen.c:2207:                RPC_SET_OPT(rbxl, SSL_CERT_DEPTH_OPT, "ssl-cetificate-depth",
17:37 uebera|| xlators/mgmt/glusterd/src/glusterd-volgen.c:2777:        RPC_SET_OPT(xl, SSL_CERT_DEPTH_OPT, "ssl-cetificate-depth", goto err);
17:37 uebera|| xlators/mgmt/glusterd/src/glusterd-volgen.c:5355:        RPC_SET_OPT(xl, SSL_CERT_DEPTH_OPT, "ssl-cetificate-depth", return -1);
17:38 JoeJulian Ugh, I hate reading logs in IRC.
17:38 uebera|| Will see whether changing this solves my problem with the certificates and file a bug report afterwards.
17:38 glusterbot https://bugzilla.redhat.com/en​ter_bug.cgi?product=GlusterFS
17:39 uebera|| That's not the logs I was mentioning regarding the SSL problems. But it may point to them. If the problem persists, I'll paste the logs elsewhere ;)
17:39 JoeJulian yeah, that's an interesting find.
17:39 uebera|| Recompiling the sources for further tests...
17:40 kkeithley oh, I think it's worse than that.  in .../rpc/rpc-transport/src/socket.c, there's #define SSL_CERT_DEPTH_OPT  "transport.socket.ssl-cert-depth"
17:40 kkeithley I suspect we're not even close
17:41 uebera|| Oh. Looks it's not used often, I guess? :(
17:41 kkeithley there's a regression test for it, believe it or not.
17:41 uebera|| If it passes, you just found the next bug. :p
17:47 glugg_ joined #gluster
17:47 rafi joined #gluster
17:47 tswartz left #gluster
17:48 JoeJulian https://career4.successfactors.com/sfcareer​/jobreqcareer?jobId=6921&amp;company=iodata
17:48 JoeJulian Ugh, I hate pages that glusterbot can't read.
17:49 glugg_ hello - I just have some hell with a replica 3 host I just reinstalled. I cant add it again to the 3 host replica, as it changed its UUID. The other two hosts show this host as "State: Peer Rejected (Disconnected)"
17:49 ahino joined #gluster
17:50 bwerthmann joined #gluster
17:52 JoeJulian stop glusterd on the reinstalled host. Get the uuid from one of the peers (it's in /var/lib/glusterd/peers). Change /var/lib/glusterd/glusterd.info to use that uuid.
17:53 glugg_ @JoeJulian: Thanks for the answer! I tried, didn't work
17:54 glugg_ @JoeJulian: I'll try again, just to be sure
17:57 JoeJulian Well, that'll solve the uuid problem. There may be others.
17:57 glugg_ Ahh - now at least it's "connected" but still "rejected".
17:58 JoeJulian My guess would be vols hash mismatch
17:59 JoeJulian quickest solution is to rsync /var/lib/glusterd/vols/ from a good peer.
17:59 glugg_ or should I try detaching, removing and readding?
18:02 JoeJulian I wouldn't, but I'm lazy.
18:02 glugg_ oky :) thanks
18:07 glugg_ @JoeJulian: You've got another minute?
18:08 JoeJulian They're dwindling, steadily.
18:09 glugg_ Detaching SRV1 (reinstalled, still in peer list) means removing bricks:
18:09 JoeJulian correct
18:09 glugg_ [root@SRV3 peers]# gluster volume remove-brick data replica 2 SRV1:/gluster/data/brick1 force
18:10 glugg_ that means having now a replica 2
18:10 JoeJulian It does
18:10 glugg_ Error: volume remove-brick commit force: failed: number of bricks provided (1) is not valid. need at least 2 (or 2xN)
18:10 glugg_ ok
18:10 JoeJulian How many total bricks do you have?
18:11 glugg_ basically for this volume just 1
18:11 glugg_ Next I tried: [root@SRV3 peers]# gluster volume remove-brick data replica 3 SRV1:/gluster/data/brick1 force
18:11 glugg_ This gives: "volume remove-brick commit force: failed: given replica count (3) option is more than volume data's replica count (2)"
18:11 glugg_ So neither is ok wth
18:11 JoeJulian If you only have 1 brick, you have no replication.
18:12 JoeJulian @pasteinfo
18:12 glusterbot JoeJulian: Please paste the output of "gluster volume info" to http://fpaste.org or http://dpaste.org then paste the link that's generated here.
18:14 glugg_ sry sure 3 bricks (now 2)
18:15 level7_ joined #gluster
18:15 glugg_ http://paste.fedoraproject.org/359625/61608127/
18:15 glusterbot Title: #359625 Fedora Project Pastebin (at paste.fedoraproject.org)
18:15 chirino_m joined #gluster
18:17 JoeJulian So that server isn't in the data volume, you've already removed it.
18:17 JoeJulian It is in the engine volume so if removing it is what makes you happy, you'll need to remove the brick from there first.
18:19 glugg_ Here again comes the crazy *hit, when I try to remove the brick:
18:19 glugg_ [root@SRV3 peers]# gluster volume remove-brick data replica 3 SRV1:/gluster/data/brick1 force volume remove-brick commit force: failed: given replica count (3) option is more than volume data's replica count (2)
18:19 glugg_ and next one
18:19 glugg_ [root@SRV3 peers]# gluster volume remove-brick data replica 2 SRV1:/gluster/data/brick1 force
18:19 glugg_ volume remove-brick commit force: failed: number of bricks provided (1) is not valid. need at least 2 (or 2xN)
18:20 glugg_ wth
18:23 glugg_ I simply can't remove the SRV1 brick
18:24 glugg_ Because it's "rejected"? That comes back to henn/egg
18:27 post-factum glugg_: I'd make peer "connected" first
18:29 post-factum glugg_: you could either sync volumes metadata or try to use built-in sync command
18:29 glugg_ I just got "peer probe: failed: Probe returned with Der Socket ist nicht verbunden" on peer probing
18:29 brandon joined #gluster
18:29 post-factum umm, could you please use LANG=C instead :)?
18:30 post-factum i do not speak German
18:30 glugg_ "socket is not connected" ;)
18:30 post-factum "der socket" is pretty clear :D
18:30 glugg_ hihi
18:30 post-factum what "gluster peer status" shows?
18:31 glugg_ on the "new" SRV1: number of peers 0
18:31 post-factum but you have your volume on another server?
18:31 glugg_ on the other two: 2 peers (and 1 rejected SRV1)
18:31 glugg_ yes
18:32 post-factum and you'd like to remove srv1 completely?
18:32 glugg_ well, I would like to reconnect it, as I had to reinstall the host
18:32 glugg_ detaching SRV1 from SRV2 doesn't work, so I try to remove brick, doesn't work either
18:32 post-factum isn't "gluster peer detach" sufficient for that?
18:33 glugg_ ditp
18:33 glugg_ dito
18:33 post-factum ok, uuid substituting didn't help either?
18:33 glugg_ I did, did just help that SRV1 is now connected, but still rejecteed
18:33 post-factum it is ok
18:34 post-factum now do this:
18:34 post-factum stop gluster on srv1 and copy everything from srv2 (/var/lib/glusterd/vols) to srv1
18:34 post-factum then start gluster on srv1
18:34 post-factum remember making backups
18:35 julim joined #gluster
18:35 glugg_ That's what JoeJulian told me like 15 minutes ago ;)
18:35 post-factum i didn't scroll logs that much :)
18:35 glugg_ It's a virgin cluster, no data
18:35 post-factum i mean, backups of /var/lib/glusterd/vols on srv1
18:35 glugg_ I'll do it. I just thought I could make it the "proper" way
18:35 post-factum those are volumes metadata
18:36 post-factum the proper way is this:
18:36 glugg_ there is NO data inthere
18:36 post-factum gluster volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
18:36 post-factum but if that doesn't work, manual copy should do the trick
18:37 post-factum who cares about data here ;)
18:37 glugg_ haha..
18:38 glugg_ volume sync: failed: SRV2, is not a friend
18:38 glugg_ ?? not on FB
18:39 glugg_ Will I get it right the "proper way"? :)
18:39 post-factum i doubt, try to do that manually
18:40 post-factum friendship is hard nowadays
18:40 JoeJulian It's all about good communication.
18:40 glugg_ ok, thanks :)
18:40 glugg_ yep, thanks again, also to @JoeJulian
18:41 JoeJulian srv1# rsync -a --delete srv2:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/
18:42 post-factum JoeJulian: brave man sync, v1.0
18:42 Pupeno joined #gluster
18:42 Pupeno joined #gluster
18:43 ahino joined #gluster
18:44 glugg_ :)
18:44 glugg_ but I'm trapped "Job for glusterd.service failed because the control process exited with error code. See "systemctl status glusterd.service" and "journalctl -xe" for details"
18:45 JoeJulian If you can't find the reason in /var/log/glusterfs/etc-glusterfs-glusterd.log, try just running it in the foreground with `glusterd --debug`
18:46 JoeJulian btw... next time don't remove the failed server. Just replace it, fix the uuid to match what it was before the reinstall, start glusterd and probe it.
18:48 glugg_ I tried that, but prhsp I meesed up something
18:48 glugg_ Anyway, it's still about friendship, as post-factum said: "[glusterd-peer-utils.c:133:glus​terd_peerinfo_find_by_hostname] 0-management: Unable to find friend: SRV2"
18:48 JoeJulian hostname resolution?
18:48 glugg_ no prob
18:48 JoeJulian firewall?
18:49 post-factum @ports
18:49 glusterbot post-factum: glusterd's management port is 24007/tcp (also 24008/tcp if you use rdma). Bricks (glusterfsd) use 49152 & up. All ports must be reachable by both servers and clients. Additionally it will listen on 38465-38468/tcp for NFS. NFS also depends on rpcbind/portmap ports 111 and 2049.
18:52 glugg_ That's what I have (also on the 2 other hosts, and there it works)
18:52 glugg_ http://paste.fedoraproject.org/359637/61032714/
18:52 glusterbot Title: #359637 Fedora Project Pastebin (at paste.fedoraproject.org)
18:53 glugg_ I already hesitated, because I didnt open 24007
18:53 glugg_ but it worked with the other two
18:53 JoeJulian Check iptables-save on the other two. My guess is they're empty.
18:55 glugg_ They're exactly the same on SRV1 and SRV2 argh
18:56 glugg_ http://paste.fedoraproject.org/359639/61060114/
18:56 glusterbot Title: #359639 Fedora Project Pastebin (at paste.fedoraproject.org)
18:57 glugg_ I'll try to reboot...
18:57 glugg_ shame on me
19:00 rwheeler joined #gluster
19:05 glugg_ didn't help
19:06 glugg_ (thats what we all thought)
19:09 JoeJulian As you can see from your paste, the ports required are open. I know some work was done to support firewalld. I'm not sure where that work is at though.
19:12 jiffin1 joined #gluster
19:14 glugg_ ok, thanks. I'll redeploy that host. And if that doesn't work, I'll redeploy all 3 ;)
19:17 JoeJulian You must have a background in Dell customer service. ;)
19:21 glugg_ not at all - I wouldn't be ashamed, if I had ;)
19:21 post-factum i guess you have smth to tell about dell service, JoeJulian
19:21 JoeJulian They just like to have you reinstall if anything goes wrong.
19:21 post-factum JoeJulian: we have 3 dell servers for google cache installed by google
19:22 post-factum they have usb flash with base os
19:22 JoeJulian Not enterprise support, mind you.
19:22 post-factum and google just reinstall everything from usb flash :)
19:22 JoeJulian :D
19:22 post-factum that is how real enterprise begins
19:23 glugg_ Well, my deployment server is VM which does not work until I have my hyperconverged oVir en working ;)
19:23 JoeJulian we pxe boot instead of usb
19:24 JoeJulian ... though we do have thee tor machines that are usb bootstrapped. No layer 2 between racks.
19:24 JoeJulian s/thee/three/
19:24 glusterbot What JoeJulian meant to say was: ... though we do have three tor machines that are usb bootstrapped. No layer 2 between racks.
19:30 glugg_ firewalld disabled on all 3 hosts, still "Unable to find friend"
19:31 post-factum are hosts defined by hostnames or by ip?
19:34 glugg_ no DNS, /etc/hosts, hostnames
19:35 glugg_ This didn't work either: for Q in `gluster volume list`; do    gluster reset $Q done
19:35 glugg_ but it's clear, as long as glusterd doesn't start on SRV1
19:37 glugg_ ok - I deleted the /var/lib/glusterd/vols entries, now glusterd starts
19:38 glugg_ so the rsync brave man job has a hickup
19:38 uebera|| JoeJulian: Filed https://bugzilla.redhat.co​m/show_bug.cgi?id=1330292 which mentions the aforementioned ssl certificate related option typos; the excerpts also contain the (persistent) auth errors. Enough for the day... ;)
19:38 glusterbot Bug 1330292: high, unspecified, ---, bugs, NEW , Inconsistent/mistyped option names "ssl-cert-depth", "ssl-cetificate-depth"
19:47 glugg_ wth... it just worked
19:49 post-factum unlikely
19:49 post-factum every consequence has the reason
19:50 glugg_ deleted SRV/vols/* ; deleted /var/lib/glusterd (without glusterd.info) ; relaunched glusterd ; peer probe SRV2 - done
19:51 glugg_ deleted SRV1/vols/* ; deleted SRV1/var/lib/glusterd (without glusterd.info) ; relaunched glusterd ; peer probe SRV2 - done
19:52 glugg_ perhaps the "gluster reset $vols" ?
19:53 glugg_ anyway - how much time does it take to replicate the vols? Still nothing in /var/lib/glusterd/vols
19:54 post-factum should be instant action
19:54 glugg_ argh
19:55 post-factum we talk not about replicating the volume but about propagating volume metadata
19:55 post-factum even metametadata
19:55 glugg_ still "no volumes present"
19:55 post-factum gluster peer status
19:55 post-factum :)
19:55 glugg_ it's ok, as said - see the other 2 hosts
19:56 glugg_ finally :)
19:56 glugg_ State: Accepted peer request (Connected)
19:56 post-factum that is not the desired state
19:56 glugg_ hmpf
19:57 post-factum should be "State: Peer in Cluster (Connected)" everywhere
19:57 glugg_ ok..
20:00 glugg_ It now is - after restarting the daemon on the other 2 peers
20:00 glugg_ synced
20:01 glugg_ damn, it's time for a glas whatever now
20:04 mowntan joined #gluster
20:08 JoeJulian Thanks for helping out the community, uebera||
20:08 JoeJulian uebera||++
20:08 glusterbot JoeJulian: uebera||'s karma is now 1
20:08 volga629 joined #gluster
20:09 volga629 Hello Everyone, update to 3.7.11  and SSL for management still not operational 0-socket.management:   error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
20:15 mowntan joined #gluster
20:22 JoeJulian volga629: any chance it's bug 1330292 ?
20:22 glusterbot Bug https://bugzilla.redhat.com:​443/show_bug.cgi?id=1330292 high, unspecified, ---, bugs, NEW , Inconsistent/mistyped option names "ssl-cert-depth", "ssl-cetificate-depth"
20:23 volga629 let me  check
20:24 volga629 in my case it complain about SSL method
20:25 JoeJulian Shouldn't use SSL3 anyway, imho.
20:26 post-factum isn't ssl3 disable in recent openssl versions?
20:26 post-factum s/disable/disabled/
20:26 glusterbot What post-factum meant to say was: isn't ssl3 disabled in recent openssl versions?
20:26 volga629 yes, it should use min TLSv1
20:33 Pupeno joined #gluster
20:36 volga629 I thought should have config option set  !SSLv3:!TLv1:TLv1.1:TLSv1.2
20:38 volga629 ups TLS
20:49 hackman joined #gluster
20:51 Pupeno joined #gluster
20:55 mowntan joined #gluster
20:56 nathwill does rebal-throttle apply also to heal activity in the case of non-distributed replica sets?
20:57 nathwill trying to make sure we don't starve clients for resources by converting from a single node to replica 2
20:59 post-factum i believe you need to throttle healing not rebalance
20:59 post-factum there are some knobs for that
21:04 nathwill hmmm, happen to know what those might be? closest i seem to be able to find is "disable self-healing and rate-limit your stat" from http://www.gluster.org/pipermail/g​luster-users/2014-June/017476.html
21:04 glusterbot Title: [Gluster-users] Unavailability during self-heal for large volumes (at www.gluster.org)
21:10 post-factum sudo gluster volume set help | grep heal
21:10 post-factum smth like cluster.background-self-heal-count or cluster.self-heal-window-size
21:13 cyberbootje joined #gluster
21:18 post-factum JoeJulian: btw
21:18 post-factum JoeJulian: if i have to upgrade replica 2 with 20 volumes, and would like to heal each volume one by one
21:18 post-factum JoeJulian: how could i achieve that?
21:19 post-factum JoeJulian: i mean, if there is any possibility to start volumes independently, but not all of them?
21:27 DV joined #gluster
21:35 JoeJulian post-factum: I did that. It's a bitch.
21:37 mowntan joined #gluster
21:41 post-factum JoeJulian: how?
21:45 JoeJulian The way I did it was to *not* kill the glusterfsd processes during the upgrade.
21:45 JoeJulian Then when I was ready to upgrade any specific server's volume, I would kill the glusterfsd associated with that volume and restart glusterd.
21:45 post-factum omg
21:46 JoeJulian Yeah, it wasn't pretty.
21:46 post-factum and that worked?
21:46 JoeJulian Until I had an issue with self-heal and tried a "start force".
21:46 JoeJulian That f'ed it all up.
21:46 post-factum did you kill with -15?
21:46 JoeJulian yes.
21:47 post-factum i don't want my data to be fcked up
21:47 post-factum :)
21:47 JoeJulian If you kill -9, you have clients hanging for ping-timeout.
21:47 JoeJulian I found it's easier to just down the whole server, upgrade it, and start it and wait for all the heals to complete.
21:47 JoeJulian Just make sure you turn off client-side heals.
21:48 JoeJulian cluster.data-self-heal off
21:48 JoeJulian (and all the others as far as I'm concerned)
21:48 post-factum should i set that to off before upgrade and back to on after heal is completed?
21:48 glugg_ re guys . I think i really f*cked up. fortunately no data yet.
21:49 JoeJulian Oh! If you have no data then it should be really easy to fix.
21:49 glugg_ Brick SRV2:/gluster/data/brick1   N/A       N/A        N       N/A
21:49 glugg_ I removed-brick etcpadady
21:50 glugg_ I used the trick "setfattr -x trusted.glusterfs.volume-id /gluster/data/brick1" to forget about the brick
21:51 JoeJulian Once you remove a brick, you should wipe it before adding it to a volume again.
21:51 glugg_ hmm...
21:52 glugg_ you mean rm the vol?
21:54 JoeJulian No
21:54 JoeJulian I mean the brick.
21:55 JoeJulian You removed the brick from the volume. All its metadata is no longer valid.
21:55 JoeJulian So remove everything from that brick.
21:55 glugg_ I'm really sorry, my head's spinning already
21:55 JoeJulian apparently. :D
21:55 DV joined #gluster
21:55 JoeJulian imho, you're trying to do too many operations to reach your goal.
21:55 glugg_ all I have left is the .trashcan
21:56 kihaloul joined #gluster
21:56 glugg_ well, the moment I had the connection it was ok, only my SRV1 missed in the brick replica. So I tried to get it back and f*cked up
21:57 glugg_ because I couldn't add-brick (already part of blah") I thought I'll remove it.. well
21:57 JoeJulian If you have no data, why not just wipe /var/lib/glusterd on all your servers, wipe your bricks (or just choose different directories), and start over.
21:58 glugg_ Ahhh...that's the answer I was afraid of ;)
21:58 JoeJulian It just seems easier.
21:58 glugg_ yess
21:58 JoeJulian We can talk you through fixing it, but you seem flustered and I don't think anything would stick.
21:58 glugg_ indeed
21:58 JoeJulian I'd much rather you learn than have me tell you.
21:59 glugg_ you already had a lot of patience
21:59 JoeJulian s/me tell/me *just* tell/
21:59 glusterbot What JoeJulian meant to say was: I'd much rather you learn than have me *just* tell you.
21:59 JoeJulian I'm happy to help you get better at this though if you think I can help you do that.
22:01 glugg_ yeahyeah ;) You're never to old to get lessons
22:01 JoeJulian Don't I know it.
22:01 glugg_ thanks
22:03 JesperA joined #gluster
22:07 chirino_m joined #gluster
22:13 DV joined #gluster
22:14 glugg_ Ahhhh.... a working environment now
22:14 glugg_ Thanks guys!
22:14 glugg_ a 6 hrs lesson ;)
22:15 * post-factum will have some sleep now
22:15 JoeJulian Goodnight post-factum
22:15 glugg_ do so. I'll go on now with ovirt-hosted-engine pain
22:16 glugg_ n8
22:16 JoeJulian Have fun.
22:16 glugg_ tx
22:17 wnlx joined #gluster
22:17 phycho joined #gluster
22:18 mowntan joined #gluster
22:20 post-factum night all
22:21 nathwill night, thanks for the help post-factum :)
22:24 glugg_ night post-factum
22:24 glugg_ thanks
22:26 F2Knight joined #gluster
22:29 F2Knight joined #gluster
22:38 phycho joined #gluster
22:39 F2Knight_ joined #gluster
22:43 F2Knight joined #gluster
22:57 jbrooks joined #gluster
23:03 F2Knight_ joined #gluster
23:06 nathwill hmmm, seeing lots of mailing list comments about SHD-induced resource issues during replica expansion, even with the rate-limiting options... assuming those don't work to our satisfaction, would it be insane to try a SIGSLEEP/SIGCONT loop on the SHD pid?
23:08 nathwill i know we can do this effectively with rsync, so curious if it's "safe" for SHD
23:22 nathwill er.. s/sigsleep/sigstop/
23:40 lanning I don't think that will stop client triggered self heals.
23:45 nathwill ah, but that would be on a more limited set of files, right?
23:45 nathwill just the ones the clients are actually touching
23:45 nathwill that would probably be fine; most of our files are dormant
23:46 nathwill until i can get the product owner to let us delete them :D
23:52 lanning Back them up, then delete them... see if there are any complaints... :P
23:53 nathwill ha. yes, i dream about this daily

| Channels | #gluster index | Today | | Search | Google Search | Plain-Text | summary