Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-08-25

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
01:05 lordd_ joined #fuel-dev
01:20 angdraug joined #fuel-dev
01:25 teran joined #fuel-dev
02:56 teran joined #fuel-dev
03:05 lordd_ joined #fuel-dev
03:56 teran joined #fuel-dev
04:57 teran joined #fuel-dev
05:06 lordd_ joined #fuel-dev
05:33 lordd_ joined #fuel-dev
05:50 salmon_ joined #fuel-dev
05:58 sambork joined #fuel-dev
05:58 teran joined #fuel-dev
06:19 prmtl joined #fuel-dev
06:30 dshulyak_ joined #fuel-dev
06:37 adanin joined #fuel-dev
06:46 eshumkher joined #fuel-dev
06:47 EeeBotu New bug #1361036 in fuel: "Ceph Rados GW does not support bulk operations" [High, Confirmed] https://bugs.launchpad.net/bugs/1361036
06:50 prmtl_ joined #fuel-dev
06:57 pasquier-s joined #fuel-dev
07:09 teran joined #fuel-dev
07:13 dancn joined #fuel-dev
07:16 Longgeek joined #fuel-dev
07:17 HeOS joined #fuel-dev
07:24 warpc_ joined #fuel-dev
07:32 bookwar joined #fuel-dev
07:38 sambork joined #fuel-dev
07:59 IvanBerezovskiy joined #fuel-dev
07:59 sambork joined #fuel-dev
08:03 lordd_ joined #fuel-dev
08:20 tzn joined #fuel-dev
08:26 enikanorov joined #fuel-dev
08:38 dmitryme Hey library people, here is one more fix for a critical issue in deployment in 5.1/5.0.2: https://review.openstack.org/#/c/115326 and https://review.openstack.org/#/c/116214
08:38 dmitryme as always, review is appreciated
08:49 evgeniyl mattymo: https://review.openstack.org/#/c/116260/ hmm, for some reasons CI set -1 for your patch
08:50 mattymo ok I'll just hit rebase
08:50 mattymo it's normal evgeniyl )
08:51 evgeniyl mattymo: ok, cool)
09:06 adanin joined #fuel-dev
09:08 tuvenen joined #fuel-dev
09:24 t_dmitry joined #fuel-dev
09:29 lordd_ joined #fuel-dev
09:30 teran joined #fuel-dev
09:30 lordd_ joined #fuel-dev
09:36 holser left #fuel-dev
09:44 dshulyak_ joined #fuel-dev
09:48 dancn joined #fuel-dev
09:49 saline joined #fuel-dev
09:49 salmon_ joined #fuel-dev
09:55 teran joined #fuel-dev
09:56 teran_ joined #fuel-dev
09:58 salmon_ joined #fuel-dev
10:00 sambork joined #fuel-dev
10:05 e0ne joined #fuel-dev
10:12 e0ne joined #fuel-dev
10:15 tnurlygayanov joined #fuel-dev
10:20 e0ne joined #fuel-dev
10:22 shauno_ joined #fuel-dev
10:26 e0ne joined #fuel-dev
10:26 obcecado hi guys, to build fuel iso, should i use ubuntu 12 or 14?
10:31 rvyalov obcecado: hi, 12.04
10:32 e0ne joined #fuel-dev
10:33 shauno__ joined #fuel-dev
10:39 tzn joined #fuel-dev
10:43 e0ne joined #fuel-dev
10:45 obcecado thank you rvyalov
10:46 shauno_ joined #fuel-dev
10:48 e0ne joined #fuel-dev
10:55 EeeBotu New bug #1361158 in fuel: "Galera doesn't sync after adding controllers" [High, New] https://bugs.launchpad.net/bugs/1361158
10:56 e0ne joined #fuel-dev
11:01 kaliya joined #fuel-dev
11:03 tldr_ joined #fuel-dev
11:05 tldr_ всем привет, подскажите, пожалуйста, почему на свежей установке fuel'а и загрузки нод по pxe в bootstrap, они не определяются мастером как доступные?
11:05 tldr_ где бы почитать как механизм определния работает
11:07 e0ne joined #fuel-dev
11:12 e0ne joined #fuel-dev
11:13 Kieleth joined #fuel-dev
11:15 kaliya joined #fuel-dev
11:17 warpc_ tldr_: привет. Не определяются как доступные в смысле висят как офлайн или совсем не появились в веб-интерфейсе?
11:17 Kieleth hi everybody! I'm using scripted mirantis installation, got fuel-master installed, but with no port 8000 opened, ps -ef | grep fuel gives nothing, anyone knows where to begin to troubleshoot this?
11:17 tldr_ warpc_, совсем не появились
11:17 Kieleth is this the right channel?
11:18 evg tldr_: hello, please write english here
11:18 tldr_ fuel node is also doesn't show any valid nodes
11:19 evg tldr_: you can read here about pxe boot http://docs.mirantis.com/openstack/fuel/fuel-5.0/user-guide.html#changing-pxe-network-parameters-during-installation
11:19 e0ne joined #fuel-dev
11:20 evg tldr_: do they realy boot in bootstrap?
11:20 warpc_ tldr_: If nodes really booted (you can access to it using ssh), run /opt/nailgun/bin/agent (this agent running every 5 minutes by cron define node as online/offline)
11:20 tldr_ nodes are really booted
11:20 tldr_ and are available by ip from master
11:20 tldr_ one min
11:22 teran joined #fuel-dev
11:22 [HeOS] joined #fuel-dev
11:24 warpc_ tldr_: also in node must exist file /etc/nailgun_systemtype which contain "bootstrap" (this is sign of correct node state)
11:25 dteselkin joined #fuel-dev
11:25 kaliya joined #fuel-dev
11:25 igormarnat joined #fuel-dev
11:25 tldr_ warpc_, yes, this file exists with bootstrap in it :)
11:26 warpc_ Kieleth: hi! One moment
11:28 evg tldr_: and does "fuel node" on the master node show something?
11:28 tldr_ warpc_, this is the output http://paste.openstack.org/raw/99862/
11:28 warpc_ mattymo can you help Kieleth to investigate problem with master node
11:28 tldr_ evg, as i mentioned above "<tldr_> fuel node is also doesn't show any valid nodes"
11:29 mattymo warpc_, ??
11:29 warpc_ mattymo: Kieleth: I'm using scripted mirantis installation, got fuel-master installed, but with no port 8000 opened, ps -ef | grep fuel gives nothing, anyone knows where to begin to troubleshoot this?
11:29 mattymo Kieleth, can you pastebin /var/log/puppet/bootstrap_admin_node.log
11:29 mattymo ?
11:29 evg tldr_: yes, i'm seing your logs...
11:31 e0ne joined #fuel-dev
11:35 warpc_ tldr_: do you already create a cluster?
11:35 tldr_ warpc_, no, just booted nodes in a bootstrap
11:35 tldr_ they should be available
11:36 warpc_ tldr_: please try to create simple cluster (options do not matter). This step help us to check Nailgun availability from master node
11:37 tldr_ ok, 2 min
11:39 tldr_ warpc_, i've created env via ui http://paste.openstack.org/raw/99863/
11:39 Kieleth Hi, warpc, mattymo thanks for response, unfortunately my colleague decided to cut short and reinstall, if we still have it afterwards I'll start with the bootstrap_admin_node.log
11:40 mattymo maybe it's still deploying or there was some strange issue
11:40 mattymo if 8000 isn't listening, the nginx docker container isn't running. it wouldn't run if there was a major problem with the pacakge repo or nailgun
11:42 warpc_ tldr_: good, thank you. It is looks like 8000 port (Nailgun) unavailable from external nodes.
11:43 tldr_ warpc_, is it possible to fix it?
11:46 dpyzhov xenolog: hi. do you have working solution for gateways on target nodes without public ip?
11:47 mattymo curl http://10.33.33.2:8000/api/version from a bootstrapped node or from your system
11:47 mattymo it should be reachable from any network
11:48 warpc_ tldr_: ^ please provide output for command above
11:51 tldr_ warpc_, are you talking about /opt/nailgun/bin/agent ?
11:51 tldr_ warpc_, this is the output http://paste.openstack.org/raw/99862/
11:51 warpc_ tldr_: from any undiscovered node please run `curl http://10.33.33.2:8000/api/version` and provide output
11:54 tldr_ warpc_, you're right http://paste.openstack.org/raw/99874/
11:54 e0ne joined #fuel-dev
11:55 lordd_ joined #fuel-dev
11:55 evg tldr_: dockerctl check nginx
11:55 evg tldr_: dockerctl check nailgun
11:55 mattymo tldr_, is this the IP address of your fuel master node?
11:55 mattymo 10.33.33.2?
11:55 EeeBotu New bug #1360298 in fuel: " iptables rules are missing the tcp rule for logging" [Critical, In Progress] https://bugs.launchpad.net/bugs/1360298
11:55 mattymo and try the command again from master node itself
11:56 mattymo tldr_, ^
11:56 warpc_ tldr_: if you change master node ip address, you should reboot boostrap nodes because they still use old master node ip
11:58 f13o joined #fuel-dev
11:59 tldr_ mattymo, http://paste.openstack.org/raw/99881/
11:59 tldr_ warpc_, http://paste.openstack.org/raw/99880/
12:00 tldr_ it is a clean install, and master is available to node
12:00 mattymo but not from bootstrap...
12:00 tldr_ yeah
12:01 lordd_ joined #fuel-dev
12:02 e0ne joined #fuel-dev
12:02 mattymo tldr_, what about port 8080?
12:03 mattymo 8080 and 8000 should be available on all nodes in the env
12:03 mattymo along with ssh port 22
12:04 tldr_ min
12:04 tldr_ yes, it is ok
12:05 tldr_ http://paste.openstack.org/raw/99883/
12:11 mattymo tldr_, you're being a little inspecific here. What am I looking at? from a bootstrap node? from fuel master? from your notebook?
12:12 tldr_ mattymo,  from node with bootstrap
12:12 mattymo ok so 8080 works. and http://paste.openstack.org/raw/99874/ is the output from the same node if you run curl http://10.33.33.2:8000/api/version ?
12:13 tldr_ yes, you're right
12:13 mattymo ok
12:13 mattymo that means you have some interesting iptables rules
12:13 mattymo can you pastebin iptables-save
12:13 tldr_ ok, 1 min
12:13 pasquier-s joined #fuel-dev
12:14 mattymo also I guess docker inspect (nginx container name) would help to make sure it's bound to all ports
12:15 tldr_ mattymo, http://paste.openstack.org/raw/99885/
12:17 tldr_ mattymo, http://paste.openstack.org/raw/99891/
12:18 mattymo tldr_, docker inspect 2fbbd00c8f8c
12:19 mattymo aslo try this: iptables -A FORWARD -o docker0 -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT
12:19 mattymo and see if it starts working
12:20 tldr_ this is inspect output http://paste.openstack.org/raw/99893/
12:20 tldr_ 1 min
12:21 mattymo yeah the container is bound to all interfaces (good). but I really doubt this iptables rule is causing this
12:21 tldr_ i've applied that rule and result is the same
12:21 tldr_ "curl: (7) couldn't connect to host" from node
12:22 jaypipes joined #fuel-dev
12:22 tldr_ http://paste.openstack.org/raw/99894/
12:23 mattymo I'm stumped. not sure why the connections are dropping here
12:24 tldr_ :(
12:24 e0ne joined #fuel-dev
12:24 warpc_ mattymo: tldr_ what if delete and create again nginx and nailgun containers?
12:25 tldr_ ok, i'll try
12:25 mattymo yeah might help, but I doubt it
12:25 mattymo dockerctl destroy nginx;dockerctl destroy nailgun;dockerctl start nailgun;dockerctl start nginx
12:27 warpc_ dockerctl destroy nginx; dockerctl destroy nailgun;  dockerctl build (will check all depends and start containers with correct order )
12:27 tldr_ em, ok
12:31 tldr_ http://paste.openstack.org/raw/99900/ && ctrl-c
12:31 tldr_ http://paste.openstack.org/raw/99901/
12:34 evgeniyl mattymo: is there any easy way to fix this issue https://bugs.launchpad.net/fuel/+bug/1353574 ? Is it easy to add retries? Or can we just exit from start.sh script if there was an error in puppet during keystone container start?
12:35 warpc_ tldr_: it looks fine. After it just run curl http://10.33.33.2:8000/api/version from a bootstrapped node
12:37 rmoe joined #fuel-dev
12:39 salmon_ evgeniyl: do you have logs from keystone container?
12:39 tldr_ warpc_, http://paste.openstack.org/show/99903/
12:40 mattymo evgeniyl, let me look at the provider and see if it has retry options
12:40 salmon_ ah, there are logs. evgeniyl it looks like postgres was not running
12:40 evgeniyl salmon_: I attached the logs, they should be in tarball var/log/docker-keystone.log and in directory var/log/docker-logs
12:40 tldr_ but 8080 and 80 is still available
12:41 salmon_ evgeniyl: yup, I see them. see me previous message. Keystone can not connect to postgres
12:42 mattymo it's already delaying 600s for db_sync..
12:42 * mattymo looks at pullet logs
12:42 mattymo puppet logs*
12:43 warpc_ tldr_:  mattymo: i am confused. tldr_ it will be cool, if you give us debug logs from fuel web. We try to reproduce problem and investigate it more closer.
12:44 tldr_ ok, please wait a bit
12:47 evgeniyl salmon_: mattymo yes, there might be any problem with db during the upgrade, it's is the reason why my suggestion was to add retry, or exit from the start.sh if there was a problem with `puppet apply`
12:49 salmon_ evgeniyl: but if it's problem with connecting to postgres, after retry, it can fail again
12:49 mattymo setting up admin tenant fails b/c of 500 error
12:49 mattymo it doesn't look like it's retrying
12:50 salmon_ mattymo: currently it doesn't
12:50 mattymo the keystone admin logic retries on errors 400, 111, 503, and 504
12:50 mattymo but not on 500
12:51 mattymo I can propose a patch but I don't think we have a working jenkins job that builds an upgrade tarball from a gerrit patch
12:51 evgeniyl salmon_: I went to the container and ran the command manually, it worked fine, during some time there was a problem with connectivity, but then it disappeared
12:52 salmon_ evgeniyl: yeah, we can add sleep 10 at the begining of the script ;)
12:52 dshulyak_ joined #fuel-dev
12:53 mattymo evgeniyl, if you simply start the container again, it works ?
12:54 agrebennikov joined #fuel-dev
12:54 agrebennikov mihgen, guys, please take a look at https://bugs.launchpad.net/fuel/+bug/1353348
12:56 mattymo evgeniyl, https://review.openstack.org/116605
12:57 evgeniyl mattymo: I tried to run failed command manually, it worked, and I've just restarted the container, puppet passed without any errors
12:58 mattymo evgeniyl, my best guess is the keystone service isn't ready yet (because other containers start at the same time)
12:59 mihgen agrebennikov: looking
12:59 evgeniyl mattymo: thanks for the patch
13:00 aglarendil mattymo: evgeniyl I do not think that we should retry on 500 errors
13:01 aglarendil 500 error means that something is non-temporarily wrong
13:01 salmon_ aglarendil: OperationalError: (OperationalError) could not connect to server: Connection refused
13:02 aglarendil salmon_: that's the case - mysql is dead. Why should we keep retrying ?
13:03 salmon_ aglarendil: postgres and not dead, just not ready. maybe user is not created yet
13:03 aglarendil salmon_: postgres????
13:03 salmon_ aglarendil: yup
13:03 aglarendil salmon_: for which reason on Earth are we working with postgres for openstack ????
13:03 mihgen dteselkin: igormarnat folks, can you please take a look at https://bugs.launchpad.net/fuel/+bug/1353348, see message above from agrebennikov
13:04 salmon_ aglarendil: :) it's on fuel master
13:04 mihgen dteselkin: igormarnat there at least two real users who hit this issue. Let's see if we can fix it in 5.1
13:04 aglarendil salmon_: okay, then we need to fix manifests
13:04 mihgen or at least have a workaround documented in release notes
13:05 aglarendil salmon_: if postgres is not available then start waiting for it to become ready or fix puppet ordering
13:05 aglarendil salmon_: 500 error says that everything is f**ked - there is no need to retry. you'd better try to redeploy then to wait for 120 seconds for postgres
13:10 salmon_ aglarendil: in this case it just means that keystone can not connect do db
13:10 aglarendil salmon_: let's create a separate bug and fix it
13:11 aglarendil salmon_: we need to investigate what is the reason and fix puppet manifests
13:11 salmon_ aglarendil: we can check in start.sh if we can connect do db using keystone user
13:13 aglarendil salmon_: I do not see postgres db creation in the bug log
13:13 aglarendil salmon_:  may be we should move postgres db creation to keystone container
13:13 aglarendil salmon_: or make keystone start after postgres container is finished
13:15 salmon_ aglarendil: how you will determine that postgres container is ready?
13:15 mattymo it's trying to do keystone tenant-list and gets 500 error
13:15 aglarendil salmon_: for example create an exec that checks for postgres for a while
13:16 mattymo it's way easier to retry than to add "wait for service to return successfully" because token auth is already coded into the puppet provider
13:16 mattymo than to write a fresh test
13:16 aglarendil mattymo: or just run a dockerctl check postgres
13:17 salmon_ aglarendil: we need 3 thinngs: container, db, and keystone user.
13:17 aglarendil evgeniyl: mattymo let's just use dockerctl check command
13:17 aglarendil that's all we need
13:17 aglarendil that's the reason we wrote it
13:17 aglarendil I am quite surprised that we do not use it
13:17 mattymo shocked and chagrined
13:17 evgeniyl aglarendil: no way to use dockerctl check command, it will be a hack in upgrade system
13:17 aglarendil let's just add check command for keystone container start
13:18 aglarendil evgeniyl: for which reason
13:18 evgeniyl aglarendil: why you don't want to restart puppet if it failed?
13:18 mattymo evgeniyl, reimplement our checks then in your upgrade system? they're really simple
13:18 mattymo 1-3 commands to run inside container to validate it's ready
13:19 sambork joined #fuel-dev
13:19 mattymo ideally a container should die if there are problems with puppet or the service start
13:20 tnurlygayanov hi there
13:20 tnurlygayanov i have new questions :)
13:21 tnurlygayanov I work on testing Fuel ISO fro master branches (fuel code and packages from master)
13:21 tnurlygayanov and now I can see that we have hadcoded version for many components, for example, for nailgun
13:22 tnurlygayanov and in the result we have now hardcoded versions for 5.1 release in master branch, and we have another version of packages in the repositories with packets with source code of OpenStack components from master branch
13:23 agrebennikov joined #fuel-dev
13:28 mattymo evgeniyl, https://review.openstack.org/116613
13:28 mattymo again - no upgrade tarball with gerrit so you will have to apply manually
13:34 obcecado just installed last night fuel 5.1 snapshot
13:34 obcecado and i'm unable to the page
13:34 tnurlygayanov mihgen ^^ this is blocker for testing ISO, which will based on master
13:35 salmon_ mattymo: how does it help?
13:35 mattymo salmon_, they don't want to retry keystone tenant-list(((
13:35 mihgen tnurlygayanov: what do you mean by hardcoded?
13:35 mattymo they = aglarendil and evgeniyl
13:36 mattymo instead they want keystone container to die and exit 1 if puppet apply fails
13:36 mattymo then supervisor will restart it and try again
13:36 mihgen obcecado: hi, what are you unable exactly?
13:36 obcecado oopsie sorry, for incomplete report
13:36 salmon_ mattymo: ah, supervior will restart it. ok
13:36 obcecado basically i end up seeing that cloud animation
13:36 obcecado and it won't actually load the dashboard
13:37 mihgen obcecado: Fuel UI?
13:37 obcecado yes
13:37 mihgen obcecado: what version are you running?
13:37 obcecado sec
13:37 obcecado fuel-community-master-95-2014-08-25_06-28-06.iso
13:37 obcecado this iso
13:38 obcecado downloaded from the ci infrastructure
13:38 mihgen salmon_: evgeniyl where do we have nailgun.log now so we can guide obcecado where to look? ^^
13:38 mattymo in /var/log/docker-logs/nailgun/
13:38 tnurlygayanov mihgen I mean this fail: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/77/console
13:39 tnurlygayanov mihgen we have hardcoded versions for dependencies in rpm/deb specs
13:39 obcecado is that inside a docker?
13:39 obcecado a container
13:39 tnurlygayanov we can use something like "=>0.1" instead of "==0.1"
13:39 mihgen tnurlygayanov: let's go to #fuel-osci to find folks there
13:39 evgeniyl obcecado: on the host system
13:39 tnurlygayanov ok
13:39 obcecado well, on the host system i dont have that location
13:40 obcecado inside /var/log i do not have any docker related dir
13:40 mattymo obcecado, look at /var/log/puppet/bootstrap_admin_node.log
13:41 mattymo it failed for some reason
13:41 obcecado sorry i was looking in another machine
13:41 obcecado i have these files in that path: api.log  app.log  assassind.log  receiverd.log  uwsgi.log
13:44 obcecado [root@fuel nailgun]# dockerctl shell nailgun
13:44 obcecado lxc-attach: failed to get the init pid
13:44 obcecado i suppose nailgun.log is inside the container
13:44 mattymo it's not
13:44 mattymo those are all the logs
13:44 mattymo dockerctl logs nailgun will show puppet output
13:45 obcecado well i have a few errors
13:45 obcecado should i get those uploaded to pastebin or elsewhere?
13:47 mihgen obcecado: yeah
13:48 mihgen pastebin, paste.openstack.org
13:48 obcecado ok, sec
13:50 obcecado http://paste.openstack.org/show/wc6zDfxZaKjOjGg8yObb/
13:50 obcecado here it is
13:51 obcecado fyi, im running fuel on a libvirt host, the vm was created following the rhel6 template, with vda for hdds and vio for nics
14:04 mihgen obcecado: "Could not set 'directory' on ensure: Read-only file system"
14:05 mihgen can you check your fs there?
14:05 obcecado well i have a single 50gb disk, so i guess that is inaccurate
14:05 mihgen xenolog: any progress with triaging of https://bugs.launchpad.net/fuel/+bug/1355758 ?
14:05 mihgen obcecado: puppet explicitly says it can't create a dir.. what if you try that manually
14:07 obcecado that file exists mihgen
14:08 obcecado this seems to trigger that error message: /etc/fuel/client
14:08 miroslav_ joined #fuel-dev
14:08 lordd_ joined #fuel-dev
14:09 obcecado seems permissions were too restricted
14:09 obcecado edited to match puppet's spec
14:10 obcecado is restarting all containers trough supervisorctl 'the right way' to assume new settings?
14:10 dshulyak_ joined #fuel-dev
14:12 mihgen mattymo: çan you please comment? ^^
14:12 mihgen obcecado: actually if you install master node, everything should be fine
14:12 obcecado well, seems the error persists anyways
14:12 mattymo obcecado, restart containers is dockerctl restart (containername)
14:12 mattymo dockerctl list shows all containers
14:13 mihgen obcecado: you might still have disk issues? or you chown'ed something.. ?
14:13 mattymo [1;31mError: /Stage[main]/Nailgun::Client/File[/etc/fuel/client]/ensure: change from absent to directory failed: Could not set 'directory' on ensure: Read-only file system @ dir_s_mkdir - /etc/fuel/client at 16:/etc/puppet/modules/nailgun/manifests/client.pp[0m that's new
14:13 mihgen obcecado: there are tests run against ISOs, so you can look if your ISO is good or not: https://fuel-jenkins.mirantis.com/view/ISO/
14:13 obcecado mihgen: i just chowned that file to match what was specified in client.pp
14:14 mattymo obcecado, the fastest way around this: dockerctl stop all;dockerctl destroy all.. then edit /etc/dockerctl/config
14:14 mihgen obcecado: what if you simply run echo "jskfjdfkdn" > file
14:14 mattymo find the entry for /etc/fuel volume and change :ro to :rw
14:14 mihgen in same filesystem where puppet complains
14:14 mattymo mihgen, the issue is /etc/fuel is mounted read only in containers
14:14 mattymo and I didn't realize fuelclient created files in there
14:15 obcecado all my fs are rw
14:15 obcecado i just deployed that iso, edited network interfaces trough fuelmenu
14:15 mattymo you can prove it: dockerctl shell nailgun touch /etc/fuel/foo
14:15 obcecado and found this issue, no other customizations done
14:15 obcecado [root@fuel ~]# dockerctl shell nailgun touch /etc/fuel/foo
14:15 obcecado lxc-attach: failed to get the init pid
14:16 mattymo oh yeah nailgun failed to deploy
14:16 mattymo make the change to /etc/dockerctl/config I suggested, obcecado, then dockerctl destroy nailgun;dockerctl start nailgun
14:16 mattymo and it might start working again
14:17 obcecado let me read up
14:17 obcecado 16:14 >>@mattymo<< obcecado, the fastest way around this: dockerctl stop all;dockerctl destroy all.. then edit /etc/dockerctl/config
14:17 obcecado sorry i didn't get what i should mod in that config
14:18 mattymo <mattymo> find the entry for /etc/fuel volume and change :ro to :rw
14:18 obcecado ok
14:18 mihgen mattymo: why do we need all of this magic?
14:18 mihgen it should work out of the box..
14:19 mihgen was it a random disk issue during fuel master node bootstrapping? or sequence of actions done by obcecado actually leads to an issue which should be fixed?
14:19 mattymo mihgen, I'm entirely not sure. I never saw this as a problem before. It could be that fuelclient didn't set up correctly on fuel master
14:20 mattymo and fuel master host and nailgun share the same data in /etc/fuel/client for fuel cli config
14:20 mattymo it could just be a coincidence that it broke here
14:20 obcecado well, i can redeploy and test again
14:20 obcecado if you want me to
14:20 obcecado just say the word
14:20 mihgen obcecado: I think it would be awesome, and remember all steps you do
14:20 obcecado sure
14:21 mattymo obcecado, you can try fixing this one line and see if it finishes okay.. redeploy could have the same results though
14:21 mihgen so if you face it again - you'd have logs and steps to reproduce
14:21 mihgen mattymo: that's not gonna be clear setup already..
14:21 mihgen with hacks in configs
14:21 mattymo mihgen, I don't have any way to provide a custom iso to community folks
14:21 mattymo so that's the only option right now
14:22 mihgen mattymo: why custom iso? official community iso https://fuel-jenkins.mirantis.com/view/ISO/
14:22 xarses joined #fuel-dev
14:22 mattymo mihgen, if it's broken in the official community iso, how is reinstalling going to help?)
14:22 mattymo unless obcecado made several changes to his env during deployment that caused it to break
14:22 obcecado i guarantee i did not
14:23 mihgen mattymo: it's going to help to hit a bug clearly if there is one
14:23 mattymo true
14:23 mihgen mattymo: there are 2 BVTs against community ISOs
14:23 mattymo do either of them install master node? :)
14:25 mihgen mattymo: I hope so
14:25 mattymo iirc we still install 1 time and snapshot
14:25 mihgen nurla: bookwar can you please confirm it? ^^^
14:26 nurla mihgen: of course yes
14:26 mihgen nurla: so for every BVT, we do installation of master node from scratch?
14:27 obcecado i'm redeploying the snapshot 95
14:27 obcecado seems that change did not help
14:27 bookwar mattymo: yes, all bvt tests start from scratch
14:27 obcecado let me retry with a clean install
14:27 enikanorov__ joined #fuel-dev
14:27 mattymo ok great
14:28 bookwar mattymo: by the way we have fancy status page here now: https://fuel-jenkins.mirantis.com/view/ISO/ :)
14:29 nurla mihgen: yes
14:29 warpc__ joined #fuel-dev
14:30 asheplyakov_ joined #fuel-dev
14:32 aglarendil joined #fuel-dev
14:32 bdudko joined #fuel-dev
14:32 holser joined #fuel-dev
14:33 lordd_ joined #fuel-dev
14:33 mihgen bookwar: teran yep, page is cool :)
14:33 accela-dev joined #fuel-dev
14:34 monester_ joined #fuel-dev
14:38 tzn_ joined #fuel-dev
14:38 evgeniyl mattymo: could you please take a look at my comments https://review.openstack.org/#/c/116613/ ?
14:41 mattymo evgeniyl, I can't see your pastebin. looks like it's down http://paste.openstack.org/show/99958/
14:42 mattymo if you remove --detailed-exitcodes, it will return 0 unless something failed to meet dependencies
14:42 mattymo which means something could still fail in puppet and still return 0 (not so good)
14:45 Vasyan joined #fuel-dev
14:45 zzyzx joined #fuel-dev
14:47 tzn_ joined #fuel-dev
14:48 jobewan joined #fuel-dev
14:48 mattymo evgeniyl, I see what's going wrong here with --detailed-exitcodes, but I need a better fix
14:50 bdudko joined #fuel-dev
14:50 aglarendil joined #fuel-dev
14:53 lordd_ joined #fuel-dev
14:56 lordd_ joined #fuel-dev
15:02 mattymo evgeniyl, update
15:08 shauno_ joined #fuel-dev
15:11 prmtl joined #fuel-dev
15:11 evgeniyl mattymo: thanks, will check it
15:11 lordd_ joined #fuel-dev
15:12 evgeniyl mattymo: and thanks for the explanation about puppet exit code, for me it looks like unusual behavior)
15:14 mattymo puppet should exit 1 if there were any failures
15:14 mattymo it only exits 1 if a dependency is failed or if it can't compile catalog
15:17 skolekonov joined #fuel-dev
15:24 mattymo evgeniyl, mistake in that commit... I added a new one
15:24 mattymo it's so ugly and I really should have not used bash -e mode for this special case
15:28 shauno_ joined #fuel-dev
15:28 evgeniyl mattymo: oh, ok, actually it worked for me because on my env puppet returns exit code 2 in case of success
15:28 mattymo but on restart of container it should change nothing and therefore return 0
15:28 mattymo 2 = success and changes were applied
15:43 mattymo can I get another +1? https://review.openstack.org/#/c/116613/
15:44 mattymo holser, bogdando alex_didenko ^ ?
15:44 adanin joined #fuel-dev
15:49 holser mattymo, tomorrow
15:49 holser today is a holiday in Ukraine ;)
15:57 prmtl joined #fuel-dev
15:57 holser mattymo, you have my +1
15:58 agrebennikov guys, if I need to restart haproxy on the controller, do I have to do "ip netns exec haproxy service haproxy restart"?
15:59 mattymo oh right
15:59 mattymo get off)
15:59 agrebennikov or I have to "crm_resource --resource p_haproxy --force-stop" and then force-start?
16:00 EeeBotu New bug #1361279 in fuel: "Hardcoded versions of dependencies for packages for build of ISO from master branch" [Critical, Confirmed] https://bugs.launchpad.net/bugs/1361279
16:03 dshulyak_ joined #fuel-dev
16:08 EeeBotu New bug #1361282 in fuel: "Document that we do not configure backups of database after deployment" [High, Confirmed] https://bugs.launchpad.net/bugs/1361282
16:12 xarses joined #fuel-dev
16:15 teran joined #fuel-dev
16:21 holser agrebennikov, why --force-stop ?
16:21 * holser crm resource clone_p_resource stop
16:21 holser crm resource clone_p_resource start
16:22 agrebennikov holser, what if I don't want to stop the clone?
16:23 agrebennikov holser, another option is "crm_resource --resource myResource --set-parameter target-role --meta --parameter-value Stopped"
16:23 holser p_ … stop
16:24 holser agrebennikov, Well, in this case you’ll just modify CIB
16:24 holser which is not good
16:24 holser you may ban resource using pcs resource p_… ban
16:24 holser pcs resource p_… unban
16:25 holser if you need to disable it for a while at one particular node
16:26 agrebennikov holser, the problem is that if the resource is cloned - it is impossible to operate with it with help of "crm resource". This will always influence on the clone
16:27 holser just ban it then
16:27 holser ;)
16:28 agrebennikov holser, does the "ban" command stop the resource? or it changes cib only as well?
16:28 holser yes
16:28 holser it stops
16:37 agrebennikov holser, ok, tnx
16:44 prmtl joined #fuel-dev
16:46 teran joined #fuel-dev
16:49 prmtl joined #fuel-dev
16:53 prmtl joined #fuel-dev
16:59 prmtl joined #fuel-dev
17:01 EeeBotu New bug #1361294 in fuel: "Nova-compute logs filled with neutron api connection errors" [Undecided, New] https://bugs.launchpad.net/bugs/1361294
17:01 EeeBotu New bug #1361284 in fuel: "Upgrades, cannot run upgrade second time in case of error" [High, In Progress] https://bugs.launchpad.net/bugs/1361284
17:02 prmtl joined #fuel-dev
17:34 sambork joined #fuel-dev
17:39 e0ne joined #fuel-dev
17:45 angdraug joined #fuel-dev
17:47 prmtl joined #fuel-dev
17:49 rmoe joined #fuel-dev
17:50 angdraug rmoe: are you looking into https://bugs.launchpad.net/fuel/+bug/1322230 ?
17:50 rmoe angdraug: no, I was never able to get access to that environment
17:50 rmoe angdraug: it had already been deleted when I got onto that server
17:52 angdraug please update the bug then
17:52 angdraug LP thinks you're investigating it
17:53 rmoe ok
17:53 angdraug https://bugs.launchpad.net/fuel/+bug/1360044
17:54 angdraug can you try to reproduce this one? could be nasty if its confirmed
17:54 rmoe yeah, I'll do that today
18:00 a_teem_ joined #fuel-dev
18:01 t_dmitry joined #fuel-dev
18:01 ykotko joined #fuel-dev
18:04 angdraug rmoe: added a comment to #1322230 for you...
18:05 holser left #fuel-dev
18:12 rmoe thanks
18:12 tzn joined #fuel-dev
18:20 aleksandr_null joined #fuel-dev
18:20 e0ne joined #fuel-dev
18:23 fandi joined #fuel-dev
18:25 angdraug_mobile joined #fuel-dev
18:26 e0ne joined #fuel-dev
18:29 lordd_ joined #fuel-dev
18:31 lordd_ joined #fuel-dev
18:32 prmtl joined #fuel-dev
18:38 teran joined #fuel-dev
18:47 dshulyak_ joined #fuel-dev
18:48 agrebennikov angdraug openbox rulezzzzz
18:54 angdraug agrebennikov: :)
18:55 agrebennikov btw lets tals once again about caching options rbd+libvirt in fuel
18:55 agrebennikov *talk
18:57 e0ne joined #fuel-dev
19:05 EeeBotu New bug #1361331 in fuel: "Define the set of supported use cases for VirtualBox-based trials" [High, New] https://bugs.launchpad.net/bugs/1361331
19:14 angdraug I knew you were just trying to lure me in
19:15 angdraug agrebennikov: what do you want to discuss about caching?
19:16 agrebennikov angdraug, nope)) I mean I really installed openbox on the weekends and I git really excited about it
19:16 agrebennikov angdraug, but
19:17 agrebennikov I had another conversation with AndreyA, I read a couple of lists with your friend Josh as the participant
19:18 agrebennikov angdraug, they all suggest using writeback as an option when we have rbd devices attached to the vm
19:19 agrebennikov *AndreyK, sorry
19:21 angdraug enabling writeback introduces possibility of data loss
19:22 angdraug writethrough is something we can safely enable
19:22 angdraug where did you see the discussion of this on the mailing list?
19:23 angdraug best we can do is probably "rbd cache writethrough until flush"
19:24 angdraug agrebennikov: did you raise a bug against fuel for this?
19:25 angdraug looks like "rbd cache writethrough until flush" is actually recommended by Ceph and will be default in Giant+
19:25 angdraug http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042366.html
19:26 prmtl joined #fuel-dev
19:31 agrebennikov angdraug, absolutely
19:31 agrebennikov that's what he told me today ;)
19:32 agrebennikov For Giant, we have changed the default librbd caching options to:
19:32 agrebennikov rbd cache = true
19:32 agrebennikov rbd cache writethrough until flush = true
19:34 angdraug so, fuel bug?
19:38 agrebennikov not yet.... do you want me to create the bug as well as the blueprint?
19:41 angdraug enabling cache is too trivial for a blueprint
19:49 prmtl joined #fuel-dev
19:52 agrebennikov ok
19:53 agrebennikov angdraug, but does it mean you agree with that now? ;)
19:55 angdraug yes it does
19:55 angdraug send me the link and I'll mark it confirmed and schedule for 6.0
19:55 angdraug maybe even 5.1 if mihgen doesn't object
19:55 agrebennikov angdraug, oh, why?!
19:56 agrebennikov oops, 5.1 sounds much better
19:56 angdraug ^
19:56 angdraug yeah, I'll try
19:56 angdraug we're days away from HCF so no promises
20:23 angdraug joined #fuel-dev
20:35 IlluminatiJudas joined #fuel-dev
21:07 EeeBotu New bug #1361358 in fuel: "Install of Fuel fails on RAID 1 disk (C602 drive controller)" [Undecided, New] https://bugs.launchpad.net/bugs/1361358
21:13 eshumkher joined #fuel-dev
21:36 aepifanov joined #fuel-dev
21:42 obcecado hi
21:43 obcecado just finished reinstalling fuel snapshot
21:45 obcecado same result, after deploying snapshot fuel-community-master-95-2014-08-25_06-28-06.iso the nailgun container refuses to start at boot time
21:46 obcecado if i run docketctl start nailgun i get this error message: http://paste.openstack.org/show/100128/
21:52 agrebennikov angdraug, https://bugs.launchpad.net/fuel/+bug/1361391
21:57 enikanorov_ joined #fuel-dev
22:05 apodrepniy_ joined #fuel-dev
22:08 EeeBotu New bug #1361391 in fuel: "ceph caching enable" [Undecided, New] https://bugs.launchpad.net/bugs/1361391
22:08 EeeBotu New bug #1361383 in fuel: "stable/5.0 ISO Creation Execution of '/sbin/service postgresql start' returned 1" [Undecided, New] https://bugs.launchpad.net/bugs/1361383
22:12 tuvenen_ joined #fuel-dev
22:23 kupo24z joined #fuel-dev
22:48 xarses joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary