Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-11-20

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:22 agrebennikov_ joined #fuel-dev
01:16 AlexF joined #fuel-dev
01:16 rmoe joined #fuel-dev
01:41 xarses joined #fuel-dev
02:05 amnk joined #fuel-dev
02:33 Longgeek joined #fuel-dev
02:46 pboldin joined #fuel-dev
02:52 proberts joined #fuel-dev
02:52 proberts1 joined #fuel-dev
03:36 Longgeek joined #fuel-dev
03:48 pboldin joined #fuel-dev
04:15 pboldin joined #fuel-dev
04:16 mihgen xarses: rmoe Hey folks, did you see smoke/bvt failure by any chance?
04:37 Longgeek joined #fuel-dev
04:53 mrasskazov joined #fuel-dev
04:54 bdudko joined #fuel-dev
06:01 Longgeek joined #fuel-dev
06:03 Longgeek joined #fuel-dev
06:39 EeeBotu New bug #1394466 in fuel: "Compute node confuses number of eth interfaces during deploy" [Undecided, New] https://bugs.launchpad.net/bugs/1394466
06:48 dklepikov joined #fuel-dev
06:49 prmtl joined #fuel-dev
06:58 adanin joined #fuel-dev
07:08 amnk joined #fuel-dev
07:16 dshulyak joined #fuel-dev
07:17 EeeBotu New bug #1394477 in fuel: "Fuel master deployment fails: "Error: Could not find package ntp"" [Critical, New] https://bugs.launchpad.net/bugs/1394477
07:20 sambork joined #fuel-dev
07:27 amnk joined #fuel-dev
07:27 AlexF joined #fuel-dev
07:30 e0ne joined #fuel-dev
07:47 teran joined #fuel-dev
07:49 akislitsky_ joined #fuel-dev
08:04 HeOS_ joined #fuel-dev
08:06 alexbh joined #fuel-dev
08:09 stamak joined #fuel-dev
08:10 igajsin joined #fuel-dev
08:12 amnk joined #fuel-dev
08:14 omelchek joined #fuel-dev
08:16 amnk joined #fuel-dev
08:23 EeeBotu New bug #1394487 in fuel: "xz for snapshot" [Undecided, New] https://bugs.launchpad.net/bugs/1394487
08:31 salmon_ joined #fuel-dev
08:38 hyperbaba joined #fuel-dev
08:44 sambork joined #fuel-dev
08:53 avlasov joined #fuel-dev
08:56 dshulyak mattymo: hi, do you know any way to reproduce this bug https://bugs.launchpad.net/fuel/+bug/1394466 ?
08:57 dshulyak i mean some kind of procedure that will mess up interfaces
09:01 salmon_ dshulyak do you have access to scale lab?
09:02 teran joined #fuel-dev
09:09 dshulyak salmon_: no
09:10 salmon_ dshulyak I added you to chat
09:10 dshulyak salmon_: thanks
09:17 HeOS joined #fuel-dev
09:20 amnk joined #fuel-dev
09:23 adanin joined #fuel-dev
09:27 romcheg joined #fuel-dev
09:29 amnk_ joined #fuel-dev
09:33 amnk joined #fuel-dev
09:34 dancn joined #fuel-dev
09:34 aliemieshko_ joined #fuel-dev
09:35 alexz joined #fuel-dev
09:35 asledzinskiy joined #fuel-dev
09:35 dklepikov joined #fuel-dev
09:35 mattymo dshulyak1, it must be specific to their hardware
09:35 aarefiev joined #fuel-dev
09:35 mattymo how ipmi iterates interfaces vs 2.6.32-504 kernel
09:36 dkaigarodsev joined #fuel-dev
09:36 mattymo and it's probably a wontfix but I think your team should consider it
09:38 mattymo asheplyakov, ping
09:38 mattymo asheplyakov, regarding your issue with 6.0 iso build. do you have a custom patch to l23network manifests in your local git branch of fuel-library?
09:40 sambork joined #fuel-dev
09:40 zigo mattymo: Hi there! Hey, where's the actual source for nailgun-net-check ?
09:40 zigo I believe that's my last missing dependency for that bootstrap iso (if I don't need to care about fuel-agent yet...).
09:41 Longgeek joined #fuel-dev
09:43 mattymo zigo, this python lib: https://github.com/stackforge/fuel-web/tree/master/network_checker
09:43 mattymo dshulyak1, ^ what else is in nailgun-net-check?
09:43 zigo Oh...
09:43 zigo I already packaged something as "python-network-checker".
09:44 zigo It shall be named this way, because that's what you'd do when doing "import".
09:44 dshulyak1 mattymo: zigo: there is network_checker from fuel_web, all other dependencies was listed in spec
09:44 zigo That's a Debian policy...
09:45 dshulyak1 zigo: +1
09:45 dshulyak1 i dont know why it was named nailgun-net-check
09:45 zigo Ok, so that one is already done, cool! :)
09:45 zigo I'll just maybe add a "Provide:" ...
09:46 dshulyak1 also nailgun-mcagents doesnt make sense to me, they are not directly connected to nailgun, i would prefer fuel-mcollective-agents
09:52 mattymo dshulyak1, +1 great idea
09:52 mattymo but if we rename it in 1 place, we should rename it back in the original project too
09:52 omelchek joined #fuel-dev
09:52 mattymo and keep it pending until 6.1 dev cycle opens up
09:52 dshulyak1 mattymo: alex_didenko: maybe you know any way to avoid reordering of interfaces?
09:53 aliemieshko_ hi
09:53 dshulyak1 hi
09:53 dancn hello, I still not able to get a succesful make iso on 14.04, see http://paste.openstack.org/show/135145/
09:53 mattymo dshulyak1, the people who addressed reordering of interfaces are myself and andrey korolev
09:54 dancn with 12.04 the build ends successfully
09:54 mattymo but this under the presumption that the order generated by kernel in bootstrap node is the "word of god" order
09:54 asledzinskiy joined #fuel-dev
09:54 mattymo and then nailgun preserves this order
09:54 dancn IIRC your CI uses 14.04, I am confused...
09:55 alexz joined #fuel-dev
09:55 Tatyanka_Leontov joined #fuel-dev
09:55 mattymo the starting point is to take an affected node and boot it into bootstrap mode and see if the interface order is correct compared to a non-breaking node (assuming some are correct)
09:56 mattymo dshulyak1, ^
09:56 dshulyak1 mattymo: i am seeing that interface to mac mapping is different from what was initially provided to nailgun api
09:56 mattymo nice!
09:57 mattymo was it reconfigured by the user? what changed it?
09:57 dklepikov joined #fuel-dev
09:57 aliemieshko_ when i try update 5.1 to 5.1.1 i get error: 2014-11-20 09:50:21 ERROR 24944 (cli) list index out of range
09:57 aliemieshko_ Traceback (most recent call last):
09:57 aliemieshko_ File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 172, in main
09:57 aliemieshko_ run_upgrade(parse_args(sys.argv[1:]))
09:57 aliemieshko_ File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 145, in run_upgrade
09:57 aliemieshko_ config = build_config(args.src, args.password)
09:57 aliemieshko_ File "/var/tmp/upgrade/site-packages/fuel_upgrade/config.py", line 88, in build_config
09:57 aliemieshko_ return Config(config(update_path, admin_password))
09:57 aliemieshko_ File "/var/tmp/upgrade/site-packages/fuel_upgrade/config.py", line 641, in config
09:57 dshulyak1 please use paste
09:57 aliemieshko_ host_system = get_host_system(update_path, new_version)
09:57 aliemieshko_ File "/var/tmp/upgrade/site-packages/fuel_upgrade/config.py", line 199, in get_host_system
09:57 dshulyak1 dont spam
09:57 aliemieshko_ openstack_version = sorted(openstack_versions, reverse=True)[0]
09:57 aliemieshko_ IndexError: list index out of range
09:57 aliemieshko_ ==================================================
09:57 aliemieshko_ Upgrade failed
09:57 aliemieshko_ sorry
09:58 mattymo dancn, I thought it was in dev docs, but just install the precise version of multistrap: http://packages.ubuntu.com/precise/all/multistrap/download
09:58 mattymo aliemieshko_, please don't spam the chat. use paste.openstack.org
09:58 aliemieshko_ ok, sorry
09:59 aliemieshko_ how can i update 5.1 to 5.1.1 ?
10:00 avlasov joined #fuel-dev
10:01 dancn mattymo: thanks. So the prepare-build-env.sh does not address this setup
10:02 dancn mattymo: no mention of this in the doc: http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso , maybe a patch is not yet landed?
10:02 zigo Outch !
10:02 aliemieshko_ when i run ./upgrade.sh  I get a password prompt
10:03 Tatyanka_Leontov joined #fuel-dev
10:03 mattymo dancn, look here: https://github.com/stackforge/fuel-main/blob/master/prepare-build-env.sh#L43-L52
10:03 zigo I packaged https://pypi.python.org/pypi/tasklib, thinking it was the dependency needed.
10:03 mattymo if you already had multistrap installed, it won't fix it
10:03 dshulyak1 aliemieshko_: you should ping ikalnitsky about that
10:03 zigo Though it's not. It's looking like that's another thing...
10:03 zigo Where's the sources for tasklib ?
10:04 mattymo aliemieshko_, the default admin password is "admin"
10:04 zigo mattymo: dshulyak1: ^
10:04 aliemieshko_ ok, where i can find him ?
10:04 SergK dancn: prepare-build-env.sh should fix the issue you have
10:04 mattymo zigo, dshulyak1 worked on tasklib, I believe
10:04 mattymo or someone else on his team could answer
10:05 dshulyak1 zigo: source for tasklib is also in fuel-web, but you can omit it for now, we are not using it at this moment
10:05 zigo Ah ok...
10:05 zigo Well, we now have a python-tasklib (python lib for using taskwarrior) in Debian ! :)
10:06 e0ne joined #fuel-dev
10:06 teran joined #fuel-dev
10:07 dancn mattymo: Sergk: got it: the prepare-build-env.sh fix this only in master and not in stable/5.1
10:07 zigo Could someone help me test my bootstrap hardware recovery ISO image?
10:10 mattymo dancn, can you open a bug?
10:10 mattymo we can't makea backport without a bug report
10:10 mattymo make a*
10:10 aliemieshko_ yes, i enter "admin" and after get error and "Upgrade failed"
10:10 dancn mattymo: ok, will do, thanks again
10:11 mattymo aliemieshko_, I'm sorry, but I'm not the best expert on our upgrade system. Can you open a bug and we'll try to get all the info and get it to the right folks
10:11 mattymo ?
10:12 mattymo http://bugs.launchpad.net/fuel
10:12 evgeniyl__ aliemieshko_: could you please copy-paste somewhere a full log?
10:12 evgeniyl__ aliemieshko_: also where did you get this tarball?
10:13 aliemieshko_ log : http://paste.openstack.org/show/135159/
10:14 evgeniyl__ ikalnitsky: ^ do you know what it can be?
10:14 aliemieshko_ tar: http://seed.fuel-infra.org/fuelweb-iso/fuel-community-upgrade-5.1.1-106-2014-11-19_05-39-27.tar.torrent
10:14 asheplyakov joined #fuel-dev
10:15 ikalnitsky evgeniyl__: aliemieshko_: looks like the structure is wrong. i need a tarball to make sure.
10:17 dancn mattymo: done at: https://bugs.launchpad.net/fuel/+bug/1394531
10:27 EeeBotu New bug #1394531 in fuel: "make iso fails because of outdated version of yum and yum-utils" [Undecided, New] https://bugs.launchpad.net/bugs/1394531
10:30 dshulyak1 mattymo:  after deleting cobbler system with wrong data and rebooting node-45 - order of interfaces became correct, i started provisioning, and i think it will be successful
10:31 dshulyak1 mattymo: by correct i mean - it is identical to one that is initially reported to nailgun
10:32 mattymo so who switched the data? nailgun or astute? astute tells cobbler to make the profile
10:32 mattymo cobbler doesn't do any discovery itself
10:35 dshulyak1 mattymo: from snapshot that was attached by Sergey G. - it looks like that order of interfaces is changed on node itself
10:36 mattymo nice
10:36 mattymo dancn, I made this backport: https://review.openstack.org/#/c/135902/ Can you +1 it?
10:36 dshulyak1 mattymo: i restarted provisioning, so we will know soon if this is repeatable
10:38 salmon_ joined #fuel-dev
10:40 ikalnitsky bookwar: SergK: teran: please take a look to the aliemieshko_'s tarball (community one). it's broken. it doesn't match production structure. I think that's because community jenkins doesn't use our latest approaches for building upgrade tarballs. we need to fix it asap, i believe.
10:43 bookwar ikalnitsky: yes, it is an old version upgrade tarball which needs to be updated
10:50 adanin joined #fuel-dev
10:58 dshulyak1 mattymo: second provisioning attempt was successful, do you have any ideas how to eliminate possibility of interface reordering?
11:00 salmon_ dshulyak1: mattymo the question is why it happened now? The lab is running for 2months
11:02 mattymo we can make a script to reboot nodes continuously and see if it happens after several tries
11:03 mattymo but dshulyak1 said that the data reported was the correct order
11:06 dshulyak1 mattymo: it was reported at first with correct order, but then interface ordering was changed
11:06 salmon_ did someone update udev or something? Does it matter?
11:07 dshulyak1 salmon_: we are updating udev for node before provisioning, udev content based on data reported by nailgun-agent
11:09 mattymo we update udev not before provisioning, but in a postinst script
11:10 mattymo how it works is bootstrap sends data to nailgun, nailgun creates cobbler profile and says which mac addr is on the PXE network, plus a list of mac:interface pairings. provisioning runs, sets udev up, then reboots into the "correct" order that matches what we had in bootstrap
11:10 dshulyak1 mattymo: cobbler profile created for each node before provisioning
11:10 mattymo of course
11:11 mattymo the order that is in the cobbler system profile in its metadata should match what nailgun sends it, which should match what was reported during bootstrap phase
11:11 salmon_ ok, so which part doesn't work?
11:13 dancn mattymo: your patch does not fix, maybe is the wrong one... as suggested by SergK.  I cherry picked your patch and the one mentioned into the bug report on my local copy . I will wait after a deep_clean and make iso and report back
11:14 mattymo your bug report is related to multistrap. yum version cannot possibly affect ubuntu portion of make iso
11:14 dshulyak1 salmon_: mattymo: information that was received by cobbler was already incorrect, see screenshot attached by Sergey G.
11:16 mattymo dshulyak1, I see a screenshot, but that's just 1 side. what's the correct order?
11:16 dshulyak1 http://paste.openstack.org/show/135181/
11:17 mattymo oh I see in the logs 43:08 is eth0, 43:09 eth1 and f1:e5 is eth2
11:19 sambork joined #fuel-dev
11:20 mattymo udevrules=00:25:90:e3:43:08_eth0,00:25:90:e3:43:09_eth1,0c:c4:7a:1d:f1:e4_eth2,0c:c4:7a:1d:f1:e5_eth3
11:20 gilmeir joined #fuel-dev
11:21 omelchek joined #fuel-dev
11:32 EeeBotu New bug #1394546 in fuel: "KVM doesn't track time ticks" [High, In Progress] https://bugs.launchpad.net/bugs/1394546
11:32 ddmitriev joined #fuel-dev
11:35 teran joined #fuel-dev
11:39 fandi joined #fuel-dev
12:15 Tatyanka_Leontov joined #fuel-dev
12:19 Longgeek joined #fuel-dev
12:21 pboldin joined #fuel-dev
12:38 teran joined #fuel-dev
12:38 aviramb joined #fuel-dev
12:41 mihgen bogdando: aglarendil folks, do you think we can get this backport in? I removed my -1… https://review.openstack.org/#/c/135223/
12:41 aglarendil I would merge it
12:41 aglarendil +2
12:44 mihgen thx. bogdando, holser, any objections? ^^^
12:44 bogdando mihgen, LGTM, approved for merge
12:45 mihgen cool, thx
12:46 IvanBerezovskiy thanks
12:47 mihgen aglarendil: can you get warpc to the Irc?
12:48 mihgen aglarendil: there is https://bugs.launchpad.net/fuel/+bug/1318567 on him without update for a long time..
12:48 EeeBotu New bug #1394563 in fuel: "[System tests] Need to add image-based group to system tests" [High, New] https://bugs.launchpad.net/bugs/1394563
12:48 EeeBotu New bug #1394560 in fuel: "    Adding intel_iommu=on kernel parameter on Mellanox deployment" [Undecided, New] https://bugs.launchpad.net/bugs/1394560
12:48 warpc__ joined #fuel-dev
12:50 aglarendil mihgen: warpc__ is here
12:52 mihgen warpc__: hi, I wanted to ask about  https://bugs.launchpad.net/fuel/+bug/1318567 . It's on you now, are you working on it?
12:53 warpc__ mihgen: yes, work on it right now
12:55 mihgen warpc__: cool. Please move to in progress then - so we would know you've started to work on it
12:57 mihgen mattymo: dude, any idea why https://review.openstack.org/#/c/134930/ failed on fuel ci? we need this in stable/5.1, and there it passed..
12:58 warpc__ mihgen: i will, thank you.
12:59 mihgen mattymo: can you please also take a look at https://bugs.launchpad.net/fuel/+bug/1382531? Patch is proposed to master, while bug is tartgeted to 5.1.1. Looks like we need to fix milestone in bug?
12:59 mattymo mihgen, we don't even deploy ceilometer in CI
13:00 mattymo mihgen, because we should fix and get it merged in master
13:00 mattymo first
13:00 mattymo rather than a 1-off 5.1.1 targeted patch
13:00 mattymo mihgen, focus your efforts on getting people to review it and the revert it depends on, so we can backport to 5.1.1
13:00 mattymo :)
13:00 mattymo I did my best but we have a shortage of eager reviewers
13:01 mattymo https://review.openstack.org/#/c/135295/ and https://review.openstack.org/#/c/135290/2
13:01 mihgen warpc__: do you think we can close https://bugs.launchpad.net/fuel/+bug/1377105 ? I've just merged backport to 5.1.1, but fix contains "Related-Bug", not "Closes-Bug"
13:09 dancn mihgen: thanks: the patch at https://review.openstack.org/#/c/135902/ works for me and sorry for the confusion
13:11 mattymo btw my patch failed on: eystone_user[nova]) Could not evaluate: Execution of '/usr/bin/keystone --os-auth-url http://10.108.7.3:35357/v2.0/ token-get' returned 1: Authorization Failed: Request to http://10.108.7.3:35357/v2.0/tokens timed out (HTTP 408)
13:15 Bomfunk joined #fuel-dev
13:18 mihgen igajsin: hi, are you working on https://bugs.launchpad.net/fuel/+bug/1393497 or we plan to postpone the resolution to 6.1?
13:22 mihgen mattymo: about https://bugs.launchpad.net/fuel/+bug/1382531. Yes, we should fix in master first. It means that bug must target 6.0, not only 5.1.1
13:22 mattymo ok it's low for 6.0 but we are fixing in both places
13:22 mattymo mihgen, feel free to vote on my reivews
13:22 Tatyanka_Leontov Does anybody cares about http://jenkins-product.srt.mirantis.net:8080/job/6.0.ubuntu.promo_bvt/49/consoleFull or I can get it?
13:23 mihgen I don't understand much in https://review.openstack.org/#/c/135290/2. So I would rather as alex_didenko, aglarendil to take a look and merge if it's ok
13:24 mihgen Tatyanka_Leontov: please go ahead! we need to analyse what happened. bookwar, btw, why this job doesn't send email to fuel-core?
13:25 Tatyanka_Leontov mihgen: It sent, I got it there ;)
13:27 hyperbaba I am using Fuel5.1 HA deployment. O've got duplicated messages on rabbitmq from ceilometer-compute-agent resulting with duplicated samples in mongodb. Is there a way to prevent that?
13:28 amnk joined #fuel-dev
13:32 monester_laptop joined #fuel-dev
13:32 EeeBotu New bug #1394576 in fuel: "Security group creation randomly failed with Timed out waiting for a reply to message ID" [Medium, New] https://bugs.launchpad.net/bugs/1394576
13:32 EeeBotu New bug #1394568 in fuel: "[Devops] We need to be able to switch all our 6.0 swarm tests to image-based provisioning" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1394568
13:32 EeeBotu New bug #1394565 in fuel: "[Devops] Please create two jobs in 6.0 swarm to run image-based tests" [High, New] https://bugs.launchpad.net/bugs/1394565
13:34 mihgen Tatyanka_Leontov: nope, it was sent to different email group which you are part of ;) So still waiting response from devops / teran / bookwar on this
13:34 bookwar mihgen: because that was the original agreement when we created this job
13:35 mihgen bookwar: ok. I talked to teran (believe to you too before), and I hope he passed that to you, that this should be one of the official BVTs, and by the way, the most importannt one
13:35 mihgen so it should be treated in the same way, and should have same settings as other two bvts
13:36 mihgen can we reconfigure it and rename so it matches standard naming schema as well?
13:37 bookwar it already matches standard schema
13:37 bookwar it runs test group 'promo_bvt'
13:38 ruhe fuel core, can we get reviews here https://review.openstack.org/#/c/134930/ (Reduce log level for MongoDB)? this blocks backport of the same patch to 5.1.1. this bug was reproduced on scale lab, so it's quite important to have it fixed in both 6.0 and 5.1.1
13:48 EeeBotu New bug #1394587 in fuel: "[System tests] Change search by fqdn in l3-agent rescheduling test" [Medium, New] https://bugs.launchpad.net/bugs/1394587
13:48 EeeBotu New bug #1394584 in fuel: "Ceph failed to deploy with bootstrap-osd keyring not found; run 'gatherkeys'" [Critical, New] https://bugs.launchpad.net/bugs/1394584
13:54 hop joined #fuel-dev
13:55 hop hi, can someone from fuel-library review this https://review.openstack.org/#/c/135942/ ?
13:57 e0ne hi all. do i understant correct: only packages from https://github.com/stackforge/fuel-main/blob/master/requirements-{deb, rpm}.txt list will be available on iso?
14:05 AlexF joined #fuel-dev
14:07 mihgen e0ne: hi, and their deps I assume
14:07 e0ne mihgen: thanks
14:08 apodrepniy joined #fuel-dev
14:09 mihgen hyperbaba: >>> O've got duplicated messages on rabbitmq from ceilometer-compute-agent resulting with duplicated samples in mongodb   - this is pretty weird. I've not heard about anything like that
14:09 mihgen ruhe: aglarendil: holser: any idea on it? ^^^
14:11 mattymo MiroslavAnashkin, can you authenticate with nickserv?
14:11 mihgen bogdando: you are our main logs expert, are you fine with merging https://review.openstack.org/#/c/134930/ ? blocks scale lab..
14:11 aglarendil mihgen: hyperbaba: I have never heard of such issues. It seems some kind of ceilometer logic is broken.
14:12 aglarendil ruhe: ^^
14:12 hyperbaba mihgen: here is the snip from ceilometer sample-list . http://paste.openstack.org/show/135277/
14:12 hyperbaba mihgen: duplicated samples got microtime in the timestamp field
14:13 mattymo mihgen, aglarendil please do merge https://review.openstack.org/#/c/134930/
14:13 bogdando mihgen, yes, its ready
14:14 hyperbaba aglarendil: it's a stock 5.1 deployment in HA. Nothing was changed in the conf for ceilometer.
14:14 mihgen hyperbaba: I've no idea frankly but it is worth of failing a bug I think. Even it will become Invalid at the end, we would at least know the reason why duplicated entries appear… Can you file one please? Ideally straight to https://launchpad.net/mos
14:15 kaliya joined #fuel-dev
14:15 mihgen s/failing/filing/
14:15 hyperbaba aglarendil: Two of the previous filings from other people regarding duplicate samples where marked invalid with no so goot explanations
14:15 hyperbaba s/goot/good/
14:21 pboldin joined #fuel-dev
14:23 hyperbaba mihgen: I'ts not possible that no one noticed this in their deployments?
14:26 hyperbaba Is there any mongodb guru here? I need to get rid of the duplicated samples in mongodb. The key for identifying them is that timestamp has microtime and normal records do not. How can I delete those records from mongo cli. Tried to use regexp /./ for find on timestamp field but no luck.
14:31 hyperbaba hyperbaba: it it's normal sql i will write something like this "delete from meter where timestamp like '%.%'" . I know that timestamp on mongodb is a internal data type but how can i cast it with something normal?
14:33 omelchek joined #fuel-dev
14:33 EeeBotu New bug #1394599 in fuel: "[Image Based] Cloud init reset all node files after delete cluster and deploy another one on the same nodes" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1394599
14:36 ruhe mihgen: aglarendil: ack
14:37 ruhe hyperbaba: which version of MOS are you using?
14:37 hyperbaba ruhe: 5.1
14:39 ruhe hyperbaba: i haven't seen bug reports about something similar. i'll invite our ceilometer developers here
14:40 mihgen ruhe: can it be possible that every controller gets message and puts into mongo, causing dups by it?
14:41 hyperbaba ruhe: people reported duplicated samples: https://bugs.launchpad.net/ceilometer/+bug/1295487 and  https://bugs.launchpad.net/ceilometer/+bug/1261386
14:41 hyperbaba ruhe: but in different situations
14:43 hyperbaba ruhe: and some samples are recorded once every hour for example instead in 600 secs interval
14:43 hyperbaba ruhe: wierd
14:44 ruhe mihgen: hyperbaba reported issue with duplicating entries from compute agent. we have one agent per compute
14:45 ruhe hyperbaba: can you please provide examples of duplicate entries in mongo at paste them to paste.openstack.org ?
14:46 hyperbaba ruhe: http://paste.openstack.org/show/135277/ look for the entry with microtime
14:46 hyperbaba ruhe: same emission interval same resource id
14:47 mihgen ruhe: I'm about the collector. I beileve agent sends data to queue. And we have 3 rabbitmq servers..
14:48 hyperbaba mihgen: when i set ceilometer on compute to use rabbit HA queues it get's worse.
14:50 hyperbaba mihgen: there must be some caching mechanism there , not to accept same message id inside the pooling interval more than once
14:51 pboldin joined #fuel-dev
14:53 ruhe hyperbaba: mihgen: we're dicussing this with ceilometer team internally. will get back a few moments later
14:54 hyperbaba ruhe: another point. I've seen in the ceilometer logs on compute that it sends messages around 10 minutes in the hour (I've started compute ceilometers at round time to got mrtg like data) ,But some messages are get writen into the mongo 3 minutes later. No ceilometer on compute activity at that time
14:55 hyperbaba ruhe: I'l be signing out from this computer in a couple of minutes. Will try to join the channelASAP when i get home.
14:55 ruhe hyperbaba: ok
14:55 aglarendil guys, bogdando, holser, there is an essential change https://review.openstack.org/#/c/136015/1
14:56 aglarendil for rabbitmq manifests
14:56 aglarendil this change closes bug: https://bugs.launchpad.net/fuel/+bug/1394188
14:56 holser I agree with you aglarendil
14:56 aglarendil I think we should merge it into 6.0, but I am not sure that we can afford it in 5.1.1
14:57 aglarendil but it is customer-found bug
14:57 aglarendil can we somehow assess that we can use it in our 5.1.1. release?
14:57 aglarendil bogdando: holser ^^
14:58 holser I set workflow as -1
14:58 mihgen aglarendil: no way for 5.1.1 for this change.
14:58 mihgen we've got already regressions with upstream merges..
14:59 bogdando aglarendil, this one actually should be a complete fix https://review.openstack.org/#/c/127166/ instead. But we can accept a partial sync as an emergency :)
15:00 pboldin hi all, what if I need to create a fuel environment with a preset disks?
15:00 pboldin *disks layouts
15:01 teran joined #fuel-dev
15:02 pboldin using a cli preferable
15:02 teran_ joined #fuel-dev
15:07 mihgen meow-nofer any idea how to store and reuse disk layout in yaml?  See question from pboldin ^^^
15:07 pboldin mihgen: there is disks --download but I would prefer to be able to configure disks at deployment from e.g. rally
15:18 stamak joined #fuel-dev
15:30 proberts joined #fuel-dev
15:31 aglarendil mihgen: bogdando let's then think how we can fix it in with in less disruptive manner
15:31 aglarendil what we really need to do is to add sleep/retry wrappers to rabbitmq actions
15:33 EeeBotu New bug #1394619 in fuel: "Add repository for lshell package" [High, New] https://bugs.launchpad.net/bugs/1394619
15:33 EeeBotu New bug #1394617 in fuel: "ntpd is stopped for all nodes for VBox" [High, New] https://bugs.launchpad.net/bugs/1394617
15:33 EeeBotu New bug #1394613 in fuel: "[devops] Jenkins runs tests with corrupted (not downloaded completely) iso image" [Critical, New] https://bugs.launchpad.net/bugs/1394613
15:36 aglarendil bogdando: holser ^^
15:36 holser w00t?
15:38 aglarendil It seems, that we need to add sleep/retry wrappers at least for rabbitmqctl commands in 5.1.1
15:39 hyperbaba joined #fuel-dev
15:40 aglarendil also, it seems we need to add '-q' option to rabbitmqctl commands as it breaks idempotency
15:40 hyperbaba ruhe: I'm available again
15:40 tzn joined #fuel-dev
15:47 ityaptin joined #fuel-dev
15:51 ityaptin hyperbaba: Hi! Are you still have problems with ceilometer?
15:51 ruhe hyperbaba: ityaptin is our ceilometer developer. i've shared our previous chat logs with him
15:53 hyperbaba yes. This is ongoing since deployment
15:53 prmtl joined #fuel-dev
15:53 hyperbaba ruhe: yes.
15:54 hyperbaba ityaptin: yes . Very strange problem
15:54 AlexF joined #fuel-dev
15:55 ityaptin hyperbaba: why do you think that your sample are duplicated? In logs that you have shared previously we don't see this issue.
15:56 hyperbaba ityaptin: just a sec to show you
15:59 hyperbaba ityaptin: please look at this graph : http://80.74.175.93/crazy_samples.jpg. Number of running instances for that project is constantly 2.
16:00 prmtl joined #fuel-dev
16:00 hyperbaba ityaptin: sorry for not posting to some board. For some reason it's not working for me on old kde3 to paste image on the paste image site
16:01 angdraug joined #fuel-dev
16:01 ityaptin hyperbaba: No problem. give me a second to analyze this screen.
16:01 kozhukalov joined #fuel-dev
16:04 zigo Where's the meeting ???
16:06 adanin joined #fuel-dev
16:09 ityaptin hyperbaba: Please, run this statistics command on controller. ceilometer statistics -m instance -p 3600 -a sum
16:09 ityaptin and paste output please
16:12 hyperbaba ityaptin: http://paste.openstack.org/show/135370/
16:12 mihgen meeting is at #openstack-meeting-alt
16:16 hyperbaba ityaptin: is resolution 3600 for daily graphs also ? or it's 600?
16:17 hyperbaba ityaptin: on 600 period i got even crazier values.
16:20 mihgen holser: can you pls join the meeting?
16:20 ityaptin hyperbaba: 3600 - it's hourly period for statistics aggregation. We sum's values from 'instance' samples which were polled during hour.
16:22 hyperbaba ityaptin: but the resolution on the resource usage graph looks more precise than 1 agg point per hour
16:22 hyperbaba ityaptin: when daily is used
16:23 hyperbaba ityaptin: nevertheless, 3600 aggr period also shows wrong data
16:27 avlasov left #fuel-dev
16:28 ityaptin hyperbaba:  Have your cluster been loaded more than usually at hourly statistic's peaks?
16:29 hyperbaba ityaptin: no. It's basically doing nothing. Couple of instances just standing there not doing a lot of jobs.
16:29 hyperbaba ityaptin: i even reniced the rabbitmq processes and with no effect
16:36 ityaptin hyperbaba: Could you show sample list at anomaly hour? ceilometer sample-list -q "timestamp<2014-11-17T14:33:36;timestamp>2014-11-17T13:33:24" -m instance
16:36 EeeBotu New bug #1394635 in fuel: "Incorrect work of rabbit after shutdown primary controller" [High, New] https://bugs.launchpad.net/bugs/1394635
16:37 hyperbaba ityaptin: http://paste.openstack.org/show/135383/
16:37 ityaptin hyperbaba: Thanks!
16:42 IvanBerezovskiy left #fuel-dev
16:44 hyperbaba ityaptin: as You can see gazyllion same resource_id samples :(
16:45 ityaptin hyperbaba: Analysis of your data get interesting results.  Big sets of samples with same 'resource_id' have been sent as notifications, maybe.
16:45 ityaptin hyperbaba: So it's not polling meters, I think.
16:47 adanin joined #fuel-dev
16:48 hyperbaba ityaptin: I do not expect pooling to be the culprit. It's something in messaging system. But I've also noticed some pooling warnings on compute where pooler complains that can't find disk.size or similar for migrated instances. It's unrelated to this
16:49 ityaptin hyperbaba: Maybe some anomaly  instance operations were at this moments? Or it may be issue with agent-notification and rabbitmq.
16:50 hyperbaba ityaptin: No. No one touched the system, i've ask them not to do so.
16:51 ityaptin hyperbaba: Could you make check rabbitmq logs? /var/log/rabbitmq/rabbitmq@node-<number of node>.log
16:52 ityaptin hyperbaba: And please search error messages  in this log.
16:54 hyperbaba ityaptin: i just see some rejects for murano???? AMQPLAIN login refused: user 'murano' - invalid credentials. Funny, murano has it's own rabbitmq.
16:55 ityaptin hyperbaba: No. Please search errors like 'handshake_timeout'
16:55 hyperbaba ityaptin: Got thoze
16:55 ruhe hyperbaba: murano uses rabbit to communicate with VMs. in order to restrict VMs from access to rabbit used by openstack services murano uses it's own rabbit
16:56 hyperbaba ityaptin:alot
16:56 ityaptin hyperbaba: Can you paste these errors?
16:56 hyperbaba ityaptin: but the network part is working quite well.
16:58 hyperbaba ityaptin: http://paste.openstack.org/show/135391/
16:59 ityaptin hyperbaba, yes, it's known issue maybe. https://bugs.launchpad.net/mos/+bug/1362155
17:01 amnk joined #fuel-dev
17:02 ityaptin hyperbaba: I am going to go home and will be available tomorrow.
17:03 hyperbaba ityaptin: thank You very much for your time.
17:20 proberts joined #fuel-dev
17:31 agrebennikov_ joined #fuel-dev
17:34 salmon_ joined #fuel-dev
17:36 proberts1 joined #fuel-dev
17:37 EeeBotu New bug #1394662 in fuel: "[System tests] Add more debug information to rollback test" [Medium, New] https://bugs.launchpad.net/bugs/1394662
17:40 adanin joined #fuel-dev
17:43 romcheg joined #fuel-dev
17:56 AlexF joined #fuel-dev
18:07 dshulyak joined #fuel-dev
18:11 amnk joined #fuel-dev
18:18 salmon_ joined #fuel-dev
18:23 rmoe joined #fuel-dev
18:30 subscope joined #fuel-dev
18:46 gilmeir joined #fuel-dev
18:53 angdraug_ joined #fuel-dev
19:09 e0ne joined #fuel-dev
19:15 amnk joined #fuel-dev
19:27 amnk joined #fuel-dev
19:43 xarses joined #fuel-dev
19:56 mihgen angdraug_: can you take a look at https://bugs.launchpad.net/fuel/5.1.x/+bug/1391881 please. I doubt that bug with "refactor" in its title should be considered for backporting into maintenance branch ever
19:57 jobewan joined #fuel-dev
20:00 angdraug_ I doubt bug with refactor in the title has a right to exist
20:01 angdraug_ wtf!
20:01 angdraug_ https://review.openstack.org/#/q/I51a706acb3b1d4d3f0e12ef07577e7852152888f,n,z
20:03 mihgen angdraug: bug with refactor in title is fine absolutely. But never on maintenance milestone.
20:03 mihgen you would not use bp for small refactoring, but it's also good to get some info first for what should be refactored and how and why. So bug is Ok for it in my opinion
20:04 mihgen angdraug: there was a rever, according to comment somewhere
20:04 mihgen so you've got to find revert commit somehow too ;)
20:04 angdraug no, it's ok to have a refactor commit to fix the bug, but bug title should be "this and that part of code structure causes such and such problems"
20:05 mihgen angdraug: you are too strict I think )
20:05 angdraug found the revert, phew
20:05 angdraug I'm not too strict, I'm just very "emotional" as Roman put it yesterday about non-descriptive bug descriptions
20:06 angdraug it's as bad as calling all variables in your code with single characters
20:06 angdraug a = x.b(c, e + d)
20:06 angdraug how readable is that?
20:07 amnk joined #fuel-dev
20:07 angdraug have you seen my email about overly generic bug descriptions?
20:07 angdraug it cases a mess all the time...
20:10 angdraug updated the bug: Invalid for 5.1.1, Medium for 6.0
20:32 dshulyak joined #fuel-dev
20:32 mihgen angdraug: you've got some torvalds voice under Roman's pressure ;)
20:32 mihgen angdraug: I've commented on https://bugs.launchpad.net/fuel/5.1.x/+bug/1394137 btw
20:40 tatyana joined #fuel-dev
20:45 HeOS joined #fuel-dev
20:47 angdraug at least I'm not flipping people yet :)
20:48 angdraug +1, medium/won't fix
20:51 teran joined #fuel-dev
20:56 angdraug https://bugs.launchpad.net/fuel/+bug/1394635
20:56 angdraug precisely what I was complaining about
20:56 angdraug how many bugs we've had so far that fit this bug summary?
20:57 angdraug tens? maybe even a hundred by now?
20:58 MorAle joined #fuel-dev
20:58 angdraug oh, and reported only against 5.1.1, too
20:59 mihgen joined #fuel-dev
20:59 MiroslavAnashkin joined #fuel-dev
21:00 vkramskikh joined #fuel-dev
21:10 e0ne joined #fuel-dev
21:14 monester_laptop joined #fuel-dev
21:27 amnk joined #fuel-dev
21:59 amnk joined #fuel-dev
22:00 angdraug joined #fuel-dev
22:10 amnk joined #fuel-dev
22:18 xarses joined #fuel-dev
22:21 salmon_ joined #fuel-dev
22:40 amnk joined #fuel-dev
22:41 proberts joined #fuel-dev
23:03 proberts1 joined #fuel-dev
23:45 salmon_ joined #fuel-dev
23:58 proberts joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary