Perl 6 - the future is here, just unevenly distributed

IRC log for #gluster-dev, 2014-08-19

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:44 shyam joined #gluster-dev
01:01 vimal joined #gluster-dev
01:35 bala joined #gluster-dev
01:48 ilbot3 joined #gluster-dev
01:48 Topic for #gluster-dev is now Gluster Development Channel - http://gluster.org | For general chat go to #gluster | Patches - http://review.gluster.org/ | Channel Logs - https://botbot.me/freenode/gluster-dev/ & http://irclog.perlgeek.de/gluster-dev/
01:50 topshare joined #gluster-dev
02:08 shyam joined #gluster-dev
02:12 jcope left #gluster-dev
02:25 bharata-rao joined #gluster-dev
03:35 topshare joined #gluster-dev
03:53 kanagaraj joined #gluster-dev
03:59 lalatenduM joined #gluster-dev
04:01 itisravi joined #gluster-dev
04:08 hchiramm joined #gluster-dev
04:09 pranithk joined #gluster-dev
04:14 itisravi joined #gluster-dev
04:25 bmikhael joined #gluster-dev
04:29 spandit joined #gluster-dev
04:36 Rafi_kc joined #gluster-dev
04:37 anoopcs joined #gluster-dev
04:53 Rafi_kc joined #gluster-dev
04:57 atinmu joined #gluster-dev
05:02 jiffin joined #gluster-dev
05:10 topshare joined #gluster-dev
05:11 atalur joined #gluster-dev
05:22 hagarth joined #gluster-dev
05:25 kdhananjay joined #gluster-dev
05:30 ppai joined #gluster-dev
05:33 bala joined #gluster-dev
05:34 topshare joined #gluster-dev
05:49 bala joined #gluster-dev
05:49 aravindavk joined #gluster-dev
06:04 bmikhael joined #gluster-dev
06:19 rgustafs joined #gluster-dev
06:29 nishanth joined #gluster-dev
06:42 kshlm joined #gluster-dev
06:48 raghu` joined #gluster-dev
07:03 topshare joined #gluster-dev
07:12 kanagaraj joined #gluster-dev
07:21 hagarth joined #gluster-dev
07:22 bharata-rao joined #gluster-dev
07:23 aravindavk joined #gluster-dev
08:19 aravindavk joined #gluster-dev
08:26 itisravi joined #gluster-dev
08:57 hagarth joined #gluster-dev
10:12 edward1 joined #gluster-dev
10:29 kkeithley1 joined #gluster-dev
10:46 shyam joined #gluster-dev
11:07 ira joined #gluster-dev
11:48 lalatenduM joined #gluster-dev
12:14 nishanth joined #gluster-dev
12:26 hchiramm joined #gluster-dev
12:47 JustinClift Gah.  gluster.org offline again.
12:47 * JustinClift restarts varnish
12:49 JustinClift k, website seems to be working again
12:58 _Bryan_ joined #gluster-dev
13:00 shyam joined #gluster-dev
13:12 kkeithley_ ndevos: ping. maybe you know some rpm magic?   I'm building gluster (rpms) on f20 using gcc-4.9.1 installed in /usr/local/bin. rpmbuild autodetects libasan.so.1 in /usr/local/lib64 and adds a requires libasan.so.1.
13:12 kkeithley_ And even though /usr/local/lib64/libasan.so.1 is in /etc/ld.so.cache, when I try to install the gluster rpms I get a missing requires for libasan.so.1
13:13 kkeithley_ not having much luck finding an answer in google.
13:13 kkeithley_ any ideas?
13:13 kanagaraj joined #gluster-dev
13:13 ndevos kkeithley_: the requires is on a repm-provides, if you install /usr/local/lib64/libasan.so.1 without rpm, the rpm database does not know about it
13:14 ndevos s/repm-provides/rpm-provides/
13:14 kkeithley_ hmmm. okay
13:14 ndevos kkeithley_: so you either include/bundle that library, or you build an other rpm that provides that lib
13:14 kkeithley_ and indeed, I did not install gcc-4.9.1 from an rpm
13:15 kkeithley_ 4.9.1 is not available as an rpm for f20
13:16 kkeithley_ no other way around that. :-(
13:17 ndevos maybe 4.9.1 is a simple rebuild for f20?
13:18 kkeithley_ probably
13:22 kkeithley_ building it in an rpm, and installing that rpm blows away the default gcc (4.8.x) though.
13:23 JustinClift Sounds like two VM's needed. :/
13:24 JustinClift Or docker things.  Actually that might really be a decent solution.  Humble might be able to make that work easily.
13:25 hchiramm JustinClift, yep.. u can run 2 containers :)
13:27 hagarth joined #gluster-dev
13:29 JustinClift kkeithley_: ^
13:34 kkeithley_ yeah, this is already in a vm, but still
13:34 kkeithley_ another vm is not a big deal
13:40 hchiramm k :)
13:41 kkeithley_ all it takes is time
13:43 hchiramm yep :)
13:46 rgustafs joined #gluster-dev
13:48 tdasilva joined #gluster-dev
14:06 rgustafs joined #gluster-dev
14:21 hagarth joined #gluster-dev
14:27 wushudoin joined #gluster-dev
15:22 bala joined #gluster-dev
15:40 aravindavk joined #gluster-dev
16:34 nishanth joined #gluster-dev
16:34 jobewan joined #gluster-dev
16:36 bmikhael joined #gluster-dev
18:26 Humble joined #gluster-dev
18:56 ws2k3 you guys noticed that the glusterfs website is down already a few hours ?
19:07 semiosis johnmark: ^^^
19:27 _Bryan_ joined #gluster-dev
20:02 JustinClift again?
20:02 JustinClift Dammit
20:02 * JustinClift looks
20:03 JustinClift Goddammit
20:03 JustinClift I "fixed" it earlier on today by restarting varnish
20:03 JustinClift I'll do it again, but we'll need to setup something a bit better than manual-warning-by-users
20:04 JustinClift This time I restarted httpd
20:04 JustinClift Lets see if that fixes the problem
20:10 misc do we still need varnish ?
20:16 * JustinClift has no idea
20:16 JustinClift ws2k3: Thanks for pointing the problem out. :)
20:30 ws2k3 no problem but i assumed you already knew you guys dont have nagios running or anything ?
20:30 JustinClift ws2k3: Not yet
20:31 JustinClift We're working on getting stuff updated to where it should be.  It's going to take at least a few weeks though :(
20:31 ws2k3 ah oke i noticed varnich is still failing
20:32 JustinClift On that note, I'm yum updating and rebooting download.gluster.org
20:32 semiosis JustinClift: i'm going to set up pingdom
20:32 semiosis where should the alerts go?
20:32 JustinClift semiosis: gluster-infra mailing list is probably a decent place
20:32 ws2k3 if i where you just use your cell numbers to send notification
20:32 JustinClift No
20:32 semiosis lmao
20:32 JustinClift Don't do cell numbers
20:32 ws2k3 also why you guys dont use nginx ?
20:33 ws2k3 i assume for gluster.org there are only one or two system administrators ?
20:33 JustinClift I didn't setup the www box.  I probably would have used nginx :)
20:33 semiosis JustinClift: can I get a shell/sudo account on that box?
20:33 JustinClift ws2k3: Until recently it's been 0 SysAdmins with whoever's-been-around-that-has-time-to-assist getting stuff done
20:34 JustinClift semiosis: yep
20:34 semiosis JustinClift: if i send you my ssh pubkey?
20:34 JustinClift Go for it
20:34 JustinClift "semiosis" username ok?
20:34 JustinClift Actually, let me check if you're in /etc/passwd first, just in case
20:34 JustinClift Btw, I'm rebooting download.gluster.org box atm
20:34 JustinClift Just yum updated it
20:35 ws2k3 let me know if you need help with stuff i'm not so good with glusterfs but i am system administrator a a living i manage 250 webservers for a top 200 alexa company
20:35 JustinClift ws2k3: Cool. :)
20:35 JustinClift Subscribe to gluster-infra mailing list and volunteer yourself there :)
20:35 JustinClift semiosis: Yep, you're not already on the box in /etc/passwd
20:36 JustinClift ws2k3: Are you familiar with Ansible?
20:36 ws2k3 cant :( varnish error
20:36 JustinClift Gah
20:36 JustinClift 1 sec
20:36 ws2k3 no i'm not familiar with anisible i am currently looking into ansible to learn it but i had 9999 other things to do
20:36 JustinClift I'm just rebooting the damn box.  Pretending its windows
20:37 ws2k3 also i have to say you use yum so you use fedora/suse/red hat right ? not realy familliar with that distro's we use debian/ubuntu everywhere
20:37 JustinClift ws2k3: We're strongly looking at using Ansible to chuck everything in a proper config mgmt system.  Which will help with the backups. :)
20:37 JustinClift We've got one guy in my team at Red Hat who's assigned to help us out, and he's apparently a whiz with Ansible
20:38 JustinClift ws2k3: Like you, I've been meaning to learn it for ages :)
20:38 ws2k3 i might have a good job but i just got 22 so i'm still pretty young
20:38 ws2k3 yeah we used puppet for a while but it just wasnt realy that good as everybody says in my opinion
20:39 ws2k3 i have subscribed
20:44 JustinClift Cool. :)
20:44 JustinClift ws2k3: Ping the mailing list and let ppl know you exist. :)
20:44 JustinClift Also, reply to misc's thread too if possible
20:45 ws2k3 what you mean of ping the mailing list?
20:45 JustinClift Ahh
20:46 ws2k3 the use of mailing list is pretty new to me to, i never been/discuss/contributed to an opensource project before
20:47 JustinClift I mean email the mailing list to say "Hi, I'm [joe bloggs], I spend a bit of time on the gluster-dev IRC and it turns out you guys could use some admin help.  As my day job I work at a big top 200 alexa company looking after 250+ servers.  Is there anything specific you need help with?"
20:47 JustinClift Type of thing. :)
20:47 JustinClift ws2k3: No worries
20:47 JustinClift ws2k3: Most larger open source projects live and die by their mailing lists
20:48 JustinClift ws2k3: Since we have contributors from all around the world, they're one of the few ways people can communicate effectively from pretty much any time zone
20:48 semiosis how can pingdom send to gluster-infra?  i dont think i can subscribe them
20:48 JustinClift semiosis: Any idea of the email address they'll send from?  I can manually add/whitelist it
20:49 semiosis ok hold on
20:49 ws2k3 semiosis i assume there is one mail address from the entire mailing list? or in the mail server you can make an allias maby mailmen can do this
20:49 ws2k3 cant confirm subscripsion varnish down again
20:49 JustinClift If you're not sure, just try to get it to send anything to gluster-infra.  It'll hit the moderation queue, which will let me screw with it there. :)
20:49 JustinClift Gah
20:50 JustinClift ws2k3:
20:50 JustinClift ws2k3: thx
20:50 JustinClift I'll ping misc and let him know there's something wrong with httpd/varnish, and ask him to look properly
20:50 ws2k3 was it not gluster-infra@gluster.org? you could let pingdom send the email notification there then everybody gets them
20:50 semiosis from alert@pingdom.com
20:51 ws2k3 i am unable to check it now but in which language was the gluster website written? php ?
20:52 JustinClift Using Middleman from memory
20:52 JustinClift Static website generator
20:54 semiosis JustinClift: if you can get my ssh working i'll fix the varnish/apache error
20:54 semiosis your reboot didnt fix it :(
20:55 * misc look
20:56 misc i guess it may be again a dos
20:56 misc indeed
20:56 ws2k3 reboot allmost never fixes anything
20:58 ws2k3 i know a very cheap provider that offers free ddos protection, a vps would cost around 5 euro a month there, you can also put gluster.org on cloudflare which makes it more ddos proof
21:02 JustinClift semiosis: The reboot definitely fixed it for at least a little bit of time
21:02 semiosis heh ok
21:02 semiosis put that on a cron then :)
21:02 JustinClift misc: Is the ddos just a simple dos so we can iptable them?
21:03 JustinClift ws2k3: Yeah, we might need to do something a bit more special
21:03 JustinClift No idea why anyone would want to ddos gluster.org
21:03 misc JustinClift: just did, and posted on the list
21:03 semiosis JustinClift: did you allow alert@pingdom.com to gluster-infra?
21:03 ws2k3 hehe no the dsos is not a simpel iptables :)
21:03 JustinClift But, there are definitely some strange ppl in the world
21:03 misc in fact, i do not think taht's a dos on purpose
21:03 ws2k3 the hosting provider offers until 420 gbps ddos protecsion
21:03 JustinClift misc: Thx
21:04 JustinClift semiosis: Not yet
21:04 misc ths come from the same range of ip
21:04 JustinClift misc: Cool.  Idiot-dos instead of ddos
21:04 JustinClift That's manageable :)
21:04 ws2k3 dont know where the core of gluster is managed in which country
21:04 JustinClift ws2k3: Which provider?
21:04 ws2k3 Ovh
21:04 JustinClift ws2k3: We're sponsored by Rackspace
21:05 ws2k3 ah oke that should be awesome
21:05 misc rackspace is likely in a US DC, OVH is likely a french DC
21:05 JustinClift ws2k3: https://www.ovh.co.uk ?
21:05 misc i wonder about 1) contacting upstream provider 2) set a proper logging to see what the user want
21:05 misc cause the request is on xmlrpc.php
21:05 ws2k3 ovh.nl for me ovh.co.uk if you from the uk ovh.com if you from the USA
21:05 JustinClift misc: Sounds like a script kiddie
21:05 misc and I suspect this started just after the migration
21:06 semiosis misc: brute force against the wordpress api
21:06 ws2k3 Ovh is mainly french yes but they also have a DC at the canadian border
21:07 misc semiosis: maybe :/
21:07 ws2k3 servers are starting from 2 euro's a month :)
21:07 misc well, time to learn a bit more varnish, or to kick it
21:07 JustinClift semiosis: I've just white listed the alert@pingdom email
21:08 semiosis JustinClift: i'll do a test now
21:08 JustinClift So, they should come through automatically now
21:08 misc ( not that I have anything against varnish, but I think he may not help much since we are having a static website )
21:08 ws2k3 and i think cause you have a static website then you surely gonna need nginx
21:08 semiosis misc: my guess is the intent of this attack was not DoS but brute forcing access
21:08 JustinClift misc: Agreed.  Any idea if it'll help with the WP blog pieces that are still hanging around?
21:08 semiosis it only DoSed because the server is not configured right
21:09 misc semiosis: yeah, that's also something that bother me
21:09 semiosis i'm sure we could open apache up to more threads, or configure varnish appropriately, and wouldn't even notice this
21:09 JustinClift Yeah
21:09 semiosis there were not really a huge number of req's, just poorly configured server :(
21:09 misc how come asking for /xmlrpc.php do cause issue
21:09 JustinClift Static website, even under httpd, should be reasonably capable
21:09 semiosis tying up apache threads, then varnish isnt caching as much as it could be
21:10 JustinClift Maybe we should disable php for www domain?
21:10 semiosis several options
21:10 semiosis that's a good one
21:10 ws2k3 yes but with nginx you can handle like 10 times more traffic? then with apache
21:10 semiosis or have varnish *agressively* cache the static content
21:10 semiosis so it doesnt even get to apache
21:10 misc semiosis: varnish is not caching at all :)
21:10 semiosis i'll work on that
21:10 misc afaik, there is no config
21:11 misc besides default.vcl saying "the website is on port 8080"
21:11 misc which I suspect may relay too much on varnish to do smart things, and I think that's varnish is not skynet and cannot decide much
21:11 misc ws2k3: apache would be fine I think
21:13 semiosis misc: varnish is not smart, it just does what you tell it.  i'll set it up to aggressively cache the static web site & pass the rest through
21:13 semiosis this evening
21:14 misc can you also add to reject POST on /xmlrpc.php ?
21:14 semiosis sure
21:14 misc I can learn varnish language fast, but I do not like make change without having a vcs with me :/
21:15 semiosis yep, vcl is pretty easy
21:15 semiosis we can make a git repo in /etc/varnish
21:16 misc i would prefer a proper complete solution :)
21:16 JustinClift :)
21:16 semiosis i assume this server gets backed up somehow, so no need to publish the repo anywhere
21:16 misc cause having 1 git repo per software do not scale
21:16 misc semiosis: I wouldn't assume that..
21:17 semiosis lol
21:17 semiosis someone should do something about that!
21:17 JustinClift semiosis: We started the discussion about exactly that a few days ago
21:17 JustinClift We've gotten up to the rough idea of what needs to be covered
21:18 JustinClift We haven't actually done anything for it yet though ;)
21:21 MacWinner joined #gluster-dev
21:21 misc JustinClift: we already discussed about it, that's like 90% of the work :)
21:22 JustinClift :)
21:22 semiosis reading the -infra emails about CM & backups, will consider & reply later with my thoughts.
21:22 semiosis all good stuff
21:23 semiosis misc: i see what you mean by complete solution...
21:25 misc it is not that I like people making random change in prod without knowing what and when it got changed... but well, I do not like that :p
21:25 misc ( but I do it too, killing tcp connexion is just that )
21:26 * JustinClift completely agrees
21:26 JustinClift The lack of backup makes me all kinds of nervious
21:26 JustinClift nervous
21:26 JustinClift And is probably the main reason our infra changes go so slowly.  Everything knows that if it fks up badly it'll be a Real Problem
21:29 semiosis http://www.rackspace.com/knowledge_center​/article/rackspace-cloud-backup-overview
21:29 semiosis so there's that
21:31 misc I would prefer to have backup out of rackspace
21:32 misc just to avoid the case of "someone break in rackspace and erase backup and server"
21:34 misc ( like http://www.wired.com/2012/08/a​pple-amazon-mat-honan-hacking/ )
21:35 systemonkey joined #gluster-dev
21:35 JustinClift Yeah.  Completely agree
21:35 semiosis ideally both
21:36 JustinClift Or at least, if we have it in Rackspace, then we also have it elsewhere
21:36 semiosis esp if the outside backup is complicated & expensive
21:36 semiosis i think we could get something going in rackspace pretty quick, and it's already paid for
21:36 JustinClift Rackspace backup might be good for initial backup, as it's local to most of the servers
21:36 JustinClift And we have offsite back in place as well
21:36 JustinClift eg rsync from there to something inside RH firewall maybe?
21:37 misc I am not a huge fan of having them there since that's outside of the community
21:37 misc but it also easier to do, just get a share for gluster, and set rsync
21:38 misc and we have proper hardware for backup there, we just need to know the size
21:38 JustinClift maybe as a 2nd additional backup then.  We might be able to do stuff so it gets backed up by RH IT onto real tape
21:38 JustinClift So, although not a primary or secondary backup, it's a decent backup-of-last-resort :D
21:38 misc I do not think we use tape in IT :)
21:38 JustinClift Small installation yeah
21:39 misc given we have stuff around a few tera of data for mail
21:39 misc I think that's just a copy on netapp
21:39 semiosis overkill imho.  backups in rs mirrored to rh.  if everything fails, someone in rh can produce the mirror copy?
21:39 semiosis isnt that enough?
21:40 JustinClift That's still online though.  It'd strange not to have it go to offline.  But yeah, semiosis is prob right.  Overkill
21:40 misc (someone would also say "why not use glusterfs with vm on rackspace" )
21:40 JustinClift "coz we know glusterfs" :p
21:40 misc JustinClift: it may be offline, but I do not know
21:40 semiosis glusterfs is not backups
21:40 JustinClift :)
21:40 semiosis rsnapshot on glusterfs would be though...
21:40 semiosis and excruciatingly slow
21:40 misc no, but we can have a backup server
21:40 misc and have it gluster backed up
21:41 misc even if I guess we can just take a VM and that's all , no need for gluster
21:41 misc but anyway, first step is to have a idea of what we need and the size
21:41 misc and how to access
21:41 misc then we can see where and how we get the data
21:42 JustinClift We can prob establish size pretty quickly by initially rsyncing the data sets to some central spot for a first go
21:42 semiosis tar
21:42 misc and how we restore too
21:42 JustinClift Sure
21:42 JustinClift It's nearly 11pm here atm.  I'm gunna split for the night.
21:42 JustinClift You guys keep going, I'll read it in the morning :)
21:42 misc cause if we have each server send data to the cloud backup, can we make sure that server cannot read backups from others servers, etc
21:43 misc JustinClift: i would also go to bed
21:43 misc so better discuss on the list
21:43 JustinClift k, lets continue this stuff tomorrow then on list :)
21:43 JustinClift We'll start to work out where the data for each of those bits is that I listed in the email.  eg which servers, which dirs, etc.
21:43 semiosis JustinClift: gluster-meeting tomorrow right?
21:44 JustinClift Oops, me forgot
21:44 semiosis i'll be there
21:44 JustinClift Yeah it is
21:44 JustinClift 12:00 UTC now :)
21:44 JustinClift Cool. :)
21:44 semiosis wait, 8 AM us eastern?
21:44 semiosis argh
21:44 semiosis maybe not then :(
22:10 systemonkey joined #gluster-dev

| Channels | #gluster-dev index | Today | | Search | Google Search | Plain-Text | summary