Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-11-17

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
03:20 xarses joined #fuel
03:28 Longgeek joined #fuel
04:44 Longgeek joined #fuel
04:47 ArminderS joined #fuel
04:50 alexz joined #fuel
05:37 Longgeek joined #fuel
06:14 monester_laptop joined #fuel
06:17 Longgeek joined #fuel
06:18 Longgeek_ joined #fuel
06:30 neophy joined #fuel
06:57 Longgeek joined #fuel
07:13 iscsi joined #fuel
07:28 Longgeek joined #fuel
07:31 dklepikov joined #fuel
07:33 alexbh joined #fuel
07:41 Longgeek_ joined #fuel
07:47 e0ne joined #fuel
07:54 dancn joined #fuel
08:06 pasquier-s joined #fuel
08:10 HeOS joined #fuel
08:10 azemlyanov joined #fuel
08:25 stamak joined #fuel
08:25 warpc__ joined #fuel
08:34 bhi joined #fuel
08:37 neophy joined #fuel
08:54 hyperbaba joined #fuel
09:10 ddmitriev joined #fuel
09:23 hyperbaba I have to report serious issue with HA deployment and telemetry. 3 nodes for controllers and 3 nodes for mongodb deployed. Deployment successeded in fuel gui. The end result is that samples are missing from mongodb databases producing graphs like this one http://pbrd.co/1ufpny4 . The things i've found out is that replication on mongdb is not set properly. Also when i migrated instances from one node to another (already running some instances) number
09:23 hyperbaba of good samples increased. Example http://pbrd.co/1ybyXkS
09:24 artem_panchenko_ joined #fuel
09:27 hyperbaba ceilometer not reporting any fatal error eather on compute and controller nodes
09:28 kaliya Hi hyperbaba, yes please file a bug with details on how to reproduce
09:31 e0ne joined #fuel
09:35 hyperbaba kaliya: will do. But the problem is; I've put that cloud into production (forced to do so) and now i need a quick fix for it on a live system. How can i trigger reprovisioning of mongodb nodes without killing the platform?
09:36 kaliya hyperbaba: I guess you should try to reapply some puppet manifests, what do you think evg ?
09:36 hyperbaba kaliya: can you please tell me how to do that? I'm not a puppet master you know. (old school admin doing things by hand)
09:37 evg hyperbaba kaliya I won't do it without a heavy reason.
09:37 evg hyperbaba: What is the problem with mongodb replication?
09:38 hyperbaba evg: I need to have some solution. The cloud is in production and telemetry is essential for the tenants.
09:38 alexbh2 joined #fuel
09:39 sovsianikov joined #fuel
09:39 evg hyperbaba: What's wrong with mongo?
09:39 hyperbaba evg: the problem is that i can't access rs.status() from any of the nodes . Getting "unauthorized" with ceilometer user. And i do not see any replicated data on slave nodes.
09:39 evg hyperbaba: it's not a broblem
09:40 hyperbaba evg: ceilometer data only resides on master mongo nodes. On slave there are no newer files other the one from the install date.
09:40 evg hyperbaba: you have to run rs.status() with password
09:41 hyperbaba evg: Please look at the screenshots i've pasted. The graphs have a holes in timeline.
09:42 hyperbaba evg: so not all samples are recorded. All  ceilometer processes are raning on controllers and compute. There was sam stalled iptables block rule i've cleared. Other than that all reports OK. And data is missing.
09:42 evg hyperbaba: let me 5 min, I'll tell you how to run "status" (i don't have  working env now)
09:42 hyperbaba evg: how can i do that (run rs.status with password)/
09:43 hyperbaba evg: aha ok
09:48 evg hyperbaba: echo "rs.status()" | /usr/bin/mongo --username admin --password PASSWORD --host localhost --port 27017 admin
09:48 evg hyperbaba: PASSWORD you can see in puppet.log on the master or
09:48 dkusidlo joined #fuel
09:48 hyperbaba evg: i can't find admin passord in astute file
09:50 evg hyperbaba: puppet.log on the master. grep for "mongo --username admin"
09:52 hyperbaba evg : http://paste.openstack.org/show/134041/
09:52 evg hyperbaba: in astute.conf it should be too (but i'm not sure -  don't have env)
09:52 hyperbaba evg: there are no other members beside master
09:53 evg hyperbaba: is it your rs.status()?????
09:53 hyperbaba evg: yes
09:54 evg hyperbaba: terrible. What about the other nodes?
09:55 hyperbaba evg: auth fails on other node
09:55 hyperbaba evg: is password the same?
09:55 evg hyperbaba: i think so
09:56 hyperbaba evg: puppet did not do the job
09:56 hyperbaba evg: there is no grep line you gave me in puppet.log
09:59 evg hyperbaba: sorry, it didn't work from the start?
09:59 hyperbaba evg: yes
10:00 hyperbaba evg: but i did not notice it because AOK was on fuel deployment gui, and A OK was in status of the cloud in horizon
10:00 hyperbaba evg: and i've let the system in production. Now the hell broke loose becuse telemetry not working properly killing the billing and heat orchestration
10:01 pasquier-s_ joined #fuel
10:03 hyperbaba evg: is retriggering the puppet for mongo on other nodes going to work?
10:04 evg hyperbaba: so you've got 3 controller + 3 mongo nodes?
10:04 hyperbaba evg: yes.
10:04 evg hyperbaba: I'm afraid no.
10:05 evg hyperbaba: may be you can redeploy these nodes with fule cli
10:07 evg hyperbaba: i mean "fuel node -node xx -deploy"
10:07 evg hyperbaba: but production (!)
10:10 hyperbaba evg: bump
10:11 evg hyperbaba: I'm only about mongo nodes
10:13 hyperbaba evg: When the next release is going to be out?
10:13 hyperbaba evg: I am on 5.1
10:14 evg hyperbaba: I't pitty I don't have an env. Let me try to reproduce it and we'll see what we can do.
10:14 hyperbaba evg: thank you very much for your effort
10:15 dkusidlo joined #fuel
10:17 evg hyperbaba: 3 contr, 3 mongo, HA. Anything special about your env?
10:17 hyperbaba evg: ceph for all
10:18 evg hyperbaba: ok. I think I'll do 1 controller to speed up the proccess.
10:19 hyperbaba evg: go with 3 maybe that is a trigger
10:25 evg hyperbaba: ok
10:27 monester_laptop joined #fuel
10:34 hop joined #fuel
10:35 hop joined #fuel
10:36 kozhukalov joined #fuel
10:37 alexbh joined #fuel
10:38 hyperbaba I've been seing this in the ceilometer-all log:  DEBUG ceilometer.compute.pollsters.disk [-] LibvirtInspector does not provide data for  ReadBytesRatePollster get_samples /usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/disk.py:211
10:38 hyperbaba maybe this causes missed samples for telemetry
10:47 evg hyperbaba: actualy it should work on the only one mongodb node, I think
10:48 hyperbaba evg: yes. I've changed the confs to only hit master node
10:48 hyperbaba evg: but missing samples are bit more of a mistery.
10:49 evg hyperbaba: btw, check mongo service on other nodes. Are they runnig? Are db dirs exist? db.isMaster() output (should work without password)
10:50 hyperbaba evg: mongo service works on other nodes.
10:50 evg hyperbaba: db.isMaster() on the nodes?
10:51 hyperbaba evg: db.isMaster() false . isreplicaset true
10:51 hyperbaba evg: "info" : "can't get local.system.replset config from self or any seed (EMPTYCONFIG)",
10:52 cvieri joined #fuel
10:52 evg hyperbaba: aha!
10:53 hyperbaba evg: epiphany?
10:54 evg hyperbaba: check mongo configs
10:54 hyperbaba evg: what to look for/
10:54 hyperbaba evg: what to look for?
11:00 evg hyperbaba: are they existed at all?
11:01 hyperbaba evg: yes
11:01 hyperbaba evg: You mean /etc/mongodb.conf?
11:02 hyperbaba evg: it's the same on all nodes except the bind ip
11:03 evg hyperbaba: ok. Have you ran health test for ceilometer?
11:06 hyperbaba evg: there is just one naming ceilometer and it's thumbs up
11:13 pasquier-s__ joined #fuel
11:35 teran joined #fuel
11:47 hyperbaba Now i have the problem with the controllers. can't authorize cli commands like nova and ceilometer on two out of 3 controllers. Get authorisation failure
11:56 evg hyperbaba: run them with -debug. Are you sure in your network health?
11:57 evg hyperbaba: I have deployed the same env as you. Mongo is ok.
11:58 evg hyperbaba: haven't deployed controllers yet, it'll take a lot of time. But mongo nodes are ok.
12:11 pal_bth joined #fuel
12:12 teran_ joined #fuel
12:18 jaypipes joined #fuel
12:28 hyperbaba evg: system is deployed in DEBUG mode.
12:29 hyperbaba evg: network is working. Tried to reboot 2 controllers (one after another) , lost entire stack, when they come back, only 1 controller can access keystone.
12:50 pal_bth_ joined #fuel
12:56 hyperbaba I've seing this in ceilometer-all.log : WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token . Can anyone please explain me what is it releated to?
12:56 pal_bth joined #fuel
12:59 f13o joined #fuel
13:04 pal_bth_ joined #fuel
13:08 pasquier-s__ joined #fuel
13:10 pal_bth joined #fuel
13:11 vt102 joined #fuel
13:16 pal_bth_ joined #fuel
13:21 pal_bth joined #fuel
13:22 evg hyperbaba: plese try looking in keystone logs
13:30 hyperbaba evg: did not found anything. Now I see this on ceilometer-all.og on compute :  LibvirtInspector does not provide data for  OutgoingBytesRatePollster get_samples /usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/net.py:110
13:35 pal_bth_ joined #fuel
13:49 hop hi, can anyone help me with puppet manifests fuel-library?
14:25 aliemieshko_ joined #fuel
14:27 ArminderS joined #fuel
14:44 hyperbaba This ceilometer is ridiculous peace of software. All this could be done 100 times easier with simple RRD. (rant)
14:47 kaliya hyperbaba: you can install munin on servers by hand
14:47 kaliya hop: please ask in #fuel-dev
15:12 blahRus joined #fuel
15:30 rsFF la o teu buddy faz oque na lisco?
15:30 rsFF wrong window
15:30 coryc joined #fuel
15:37 angdraug joined #fuel
15:56 mattgriffin joined #fuel
15:57 jobewan joined #fuel
16:03 dkusidlo joined #fuel
16:04 iscsi joined #fuel
16:14 mnaser joined #fuel
16:25 fandi joined #fuel
16:40 dkusidlo joined #fuel
16:41 avlasov joined #fuel
17:02 mnaser Can Fuel upgrade installations (or is this something that's planned?)
17:08 iscsi mnaser: http://docs.mirantis.com/openstack/fuel/fuel-5.1/user-guide.html#upgrade-ug
17:09 mnaser iscsi: but that's upgrading fuel itself, but can fuel upgrade the underlying OpenStack installation?
17:11 iscsi mnaser: http://docs.mirantis.com/openstack/fuel/fuel-5.1/reference-architecture.html#fuel-upgrade-arch
17:11 iscsi "OpenStack engine -- Installs all data that is required for the OpenStack patching feature."
17:13 angdraug http://docs.mirantis.com/openstack/fuel/fuel-5.1/user-guide.html#update-your-openstack-environment
17:14 mnaser interesting
17:14 mnaser i'm setting up a small lab on a C6100 today
17:20 iscsi lab in virtualbox is not worse than in c6100
17:34 xarses joined #fuel
17:35 rmoe joined #fuel
17:43 mnaser iscsi: what do you mean "not worse" ?
18:03 mnaser Not sure if anyone maintain this, but it's dead
18:03 mnaser http://demo.fuel-infra.org:8000
18:07 xarses mnaser: I'll poke some one about it
18:07 mnaser cool, thanks :)
18:09 mattgriffin joined #fuel
18:10 xarses bookwar: holser mihgen ^^^
18:16 bookwar xarses: restarted nailgun there
18:18 xarses bookwar: looks good. thanks
18:18 xarses mnaser: it's up now
18:18 mnaser i guess i can play around with it until my image download finishes
18:18 mnaser thanks :)
19:03 HeOS joined #fuel
19:11 [HeOS] joined #fuel
19:43 angdraug joined #fuel
19:55 e0ne joined #fuel
20:08 monester_laptop joined #fuel
20:28 mnaser man installing fuel takes sometime...
20:51 angdraug joined #fuel
21:27 xarses joined #fuel
21:32 mnaser Just ran into an issue in Fuel
21:32 mnaser Server says "Cannot get disk parameters" (after installation)
21:42 teran joined #fuel
21:45 teran joined #fuel
21:52 mnaser Solved it, BIOS was set to "PXE boot only" .. for some reason I guess that messed it up
22:05 angdraug joined #fuel
22:30 pasquier-s__ joined #fuel
22:40 aleksandr_null joined #fuel
23:40 angdraug joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary