Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel, 2014-11-24

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:29 emagana joined #fuel
01:23 emagana joined #fuel
01:37 Rajbir joined #fuel
02:02 fandi joined #fuel
02:17 emagana joined #fuel
03:12 emagana joined #fuel
03:16 Rajbir Hi All,
03:17 Rajbir I'm trying to install Windows2012 R2 and using boot from volume but  the installation is getting stucked at device storage as it's not reading the volume disk
03:58 mattgriffin joined #fuel
04:01 ArminderS joined #fuel
04:06 emagana joined #fuel
05:21 Longgeek joined #fuel
05:54 ArminderS- joined #fuel
06:31 teran joined #fuel
06:47 adanin joined #fuel
06:53 dklepikov joined #fuel
07:03 Rajbir Hi Again,
07:03 Rajbir I'm  trying to create a snapshot for a instance using nova image-create command  and it's does showing in  GUI that the image has been created but when I  look  at /var/lib/glance/images there is no image found for the new snapshot
07:03 Rajbir also the created snapshot size showing as zero
07:04 Rajbir the openstack environment is running on Grizzly
07:05 Longgeek joined #fuel
07:05 stamak joined #fuel
07:11 sovsianikov joined #fuel
07:13 hyperbaba joined #fuel
07:18 ddmitriev joined #fuel
07:32 teran joined #fuel
07:44 emagana joined #fuel
07:52 alexbh joined #fuel
08:01 ArminderS joined #fuel
08:07 corepb joined #fuel
08:12 teran joined #fuel
08:18 e0ne joined #fuel
08:38 emagana joined #fuel
08:50 avlasov joined #fuel
08:55 adanin joined #fuel
09:20 bhi joined #fuel
09:24 HeOS joined #fuel
09:26 teran joined #fuel
09:27 akurenyshev joined #fuel
09:33 emagana joined #fuel
09:48 mrasskazov joined #fuel
09:51 Longgeek joined #fuel
10:11 ArminderS joined #fuel
10:18 azemlyanov joined #fuel
10:27 emagana joined #fuel
10:31 f13o joined #fuel
10:36 dmellado joined #fuel
10:52 ArminderS joined #fuel
11:08 teran joined #fuel
11:21 emagana joined #fuel
11:31 teran joined #fuel
11:46 monester_laptop joined #fuel
11:52 baboune joined #fuel
11:53 baboune hi, I applied https://bugs.launchpad.net/fuel/+bug/1378327 but it did not work and my disk filed up.  Now I can not login via the UI, and "fuel environment" gives an exception:     resp = self.get_raw_token_from_identity_service(**kwargs)   File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 179, in get_raw_token_from_identity_service     "%s" % e) keystoneclient.apiclient.exceptions.AuthorizationFailure: Author
11:54 baboune what can be done to restore the system?
11:59 baboune I did the obvious steps to clean up some logs under /var/log/docker-logs/remote , and dockerctl start ryslog... rebooted node... same pb
12:07 kaliya hi baboune
12:07 kaliya dockerctl logs keystone ?
12:09 kaliya baboune: but you applied patch, and the disk filled regardless? That's strange and in case is critical for us
12:10 baboune the patch is only changing one cronjob as far as I understood
12:10 baboune what are u looking for in the keystone logs?
12:11 aliemieshko_ joined #fuel
12:11 baboune outside of the keysone logs I see: "ls: cannot access /dev/sda?*: No such file or directory"
12:12 baboune kaliya: And [debug] stdcopy.go:111 framesize: 35 Traceback (most recent call last): [debug] stdcopy.go:111 framesize: 69   File "/usr/lib64/python2.6/logging/__init__.py", line 800, in emit [debug] stdcopy.go:111 framesize: 17     self.flush() [debug] stdcopy.go:111 framesize: 70   File "/usr/lib64/python2.6/logging/__init__.py", line 762, in flush [debug] stdcopy.go:111 framesize: 24     self.stream.flush() [debug] utils.go:267 [hijac
12:14 evg baboune: please check free space on the disk, then run "dockerctl check all"
12:14 baboune keystone logs: https://dl.dropboxusercontent.com/u/18136096/keystone.logs
12:15 baboune evg: [root@kds-cmc-fuel-02 ~]# df -h Filesystem           Size  Used Avail Use% Mounted on /dev/mapper/os-root  9.7G  1.5G  7.7G  17% / tmpfs               1004M     0 1004M   0% /dev/shm /dev/sda3            194M   28M  156M  16% /boot /dev/mapper/os-var    50G   15G   36G  29% /var
12:17 baboune evg: https://dl.dropboxusercontent.com/u/18136096/dockerctl-check-all.log
12:17 baboune checking container keystone No defined test for determining if keystone is ready. keystone is ready
12:17 baboune else all are ready
12:18 baboune I also see this strange logs in my console: https://dl.dropboxusercontent.com/u/18136096/FuelConsoleCapture.PNG
12:19 baboune "entering forwarding state" but it might be due to having restarted one of the docker containers
12:20 evg baboune: dockerctl list -l
12:21 baboune There is also the result of "dockerctl logs keystone > /root/keystone.log" here https://dl.dropboxusercontent.com/u/18136096/dockerctl-logs-keystone-cmd.log
12:22 baboune evg:  dockerctl list -l Name         Image                    Status       Full container name       nginx        fuel/nginx_5.1           Running      fuel-core-5.1-nginx       rabbitmq     fuel/rabbitmq_5.1        Running      fuel-core-5.1-rabbitmq    astute       fuel/astute_5.1          Running      fuel-core-5.1-astute      rsync        fuel/rsync_5.1           Running      fuel-core-5.1-rsync       keystone     fuel/keystone_5.1
12:22 baboune all are running
12:23 baboune coudl the token be corrupted? "Error: /Stage[main]/Main/Keystone_tenant[admin]: Could not evaluate: Execution of '/usr/bin/keystone --os-token 038429ff-146c-41b9-be76-2189801c854a --os-endpoint http://127.0.0.1:35357/v2.0/ tenant-get 1bc233094cab4c63895b04359e1a6b4f' returned 1: An unexpected error prevented the server from fulfilling your request. (HTTP 500)"
12:23 kaliya baboune: is fuel CLI working fine?
12:24 baboune kaliya: define fine?  it is accessible but fails with "authorization failure"
12:24 baboune fuel environment list Traceback (most recent call last):   File "/usr/bin/fuel", line 9, in <module>     load_entry_point('fuelclient==0.2', 'console_scripts', 'fuel')()   File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 105, in wrapper     handle_exceptions(exc)   File "/usr/lib/python2.6/site-packages/fuelclient/cli/error.py", line 103, in wrapper     return func(*args, **kwargs)   File "/usr/lib/python2.6/site-p
12:25 kaliya baboune: are commands `fuel nodes` etc working_
12:25 kaliya ?
12:25 baboune https://dl.dropboxusercontent.com/u/18136096/fuel-environment.log
12:26 baboune kaliya: no same error as in poster link above
12:26 baboune kaliya:  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 179, in get_raw_token_from_identity_service     "%s" % e) keystoneclient.apiclient.exceptions.AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
12:26 kaliya baboune: fuel task
12:26 baboune kaliya: nope. same
12:27 baboune kaliya: https://dl.dropboxusercontent.com/u/18136096/fuel-task.log
12:30 baboune about https://bugs.launchpad.net/fuel/+bug/1378327, according to the patch it is a simple change in  /etc/cron.hourly/logrotate where logrotate file needs to contain a "*" after 20-fuel. right?
12:31 kaliya yes
12:33 baboune yes so it did not work in my case, /var filled up
12:34 baboune this is mylogrotate: https://dl.dropboxusercontent.com/u/18136096/my-logrotate, did I make an error?
12:38 baboune any ideas/suggestions? if not my fuel is a VM, so I can try to restore it to previous snapshot, and I might be able to recover.
12:38 kaliya baboune: try to `dockerctl destroy keystone; dockerctl start keystone`
12:39 baboune kaliya: good idea
12:39 baboune but it did not work
12:41 baboune kaliya: first, I get a bad getaway then the error comes back: https://dl.dropboxusercontent.com/u/18136096/fuel-task-after-destroy-keystone.log
12:41 baboune sry  Bad Gateway (HTTP 502)
12:50 kaliya baboune: do you have something relevant errors in the keystone container logs?
12:50 vt102 joined #fuel
12:51 kaliya baboune: look if there are some DB issues recorded
12:52 baboune how cna I tell? login to the container?
12:52 kaliya look in /var/log/docker-keystone.log
12:53 baboune 2014-11-24 12:49:33.880 544 TRACE keystone.common.wsgi     raise exception.DBError(e) 2014-11-24 12:49:33.880 544 TRACE keystone.common.wsgi DBError: (InternalError) index "ix_token_expires" contains unexpected zero page at block 7 2014-11-24 12:49:33.880 544 TRACE keystone.common.wsgi HINT:  Please REINDEX it.
12:53 baboune inside the container at /var/log/keystone.log
12:55 kaliya baboune: ok, so backup the postgres with `date=$(date --rfc-3339=date); dockerctl shell postgres su postgres -c "pg_dumpall --clean' > /root/postgres_backup_$(date).sql"`
12:56 kaliya then `dockerctl shell postgres su postgres -c "psql nailgun -c 'reindex table notifications;'"`
12:56 baboune in /var/log/docker-keystone.log, I see, tail -f Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Triggered 'refresh' from 54 events Info: /Stage[main]/Keystone/Exec[keystone-manage db_sync]: Scheduling refresh of Service[keystone] Notice: /Stage[main]/Keystone/Service[keystone]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Keystone/Service[keystone]: Unscheduling refresh on Service[keystone] Notice: Finis
12:59 baboune kaliya: u sure about single quote after --clean?
13:00 kaliya baboune: no please drop it :)
13:03 baboune " dockerctl shell postgres su postgres -c "psql nailgun -c 'reindex table notifications;'" could not change directory to "/root" REINDEX"
13:03 baboune kaliya: ok?
13:05 baboune kaliya: destroy and restart keystone?
13:05 baboune or simple restart?
13:05 kaliya try to dockerctl check all before
13:05 jaypipes joined #fuel
13:07 baboune kaliya:  seems all "is ready", restart or destroy?
13:07 kaliya is the UI working?
13:08 baboune no
13:09 baboune nor is the cli... same authorization error
13:09 emagana joined #fuel
13:11 baboune same db index error
13:11 baboune DBError: (InternalError) index "ix_token_expires" contains unexpected zero page at block 7
13:11 baboune HINT:  Please REINDEX it
13:12 baboune 'INSERT INTO token (id, expires, extra, valid, user_id, trust_id) VALUES (%(id)s, %(expires)s, %(extra)s, %(valid)s, %(user_id)s, %(trust_id)s)' {'user_id': u'e5236a37816340929d664aad1dbcd910', 'extra': '{"bind": null, "token_data": {"access": {"token": {"issued_at": "2014-11-24T13:11:20.505355", "expires": "2014-11-24T14:11:20Z", "id": "2c20210239694a25a1d3a66c93d38102", "tenant": {"id": "1bc233094cab4c63895b04359e1a6b4f", "enabled":
13:13 baboune can that field be reset manually in the DB?
13:14 baboune I could delete the index and re-create it?
13:14 kaliya baboune: restart nailgun, keystone and ostf as well
13:17 baboune kaliya: same
13:18 baboune kaliya: still get  DBError: (InternalError) index "ix_token_expires" contains unexpected zero page at block 7
13:19 baboune so u want my db dump to see if u can reproduce it?  I dont know which SQL this query is and I am not familiar with postgres
13:19 kaliya nope, not needed
13:21 baboune clean up db and reimport?
13:22 kaliya try
13:22 baboune u need to guide me. I am not too familiar with PSQL
13:22 baboune kaliya: destroy postgres container and restart it?
13:23 kaliya no wait
13:24 kaliya could you reindex again, and restart the node?
13:25 baboune u sure it is: reindex table notifications;
13:25 baboune not another one?
13:25 kaliya that should be
13:25 justif_ joined #fuel
13:26 warpc___ joined #fuel
13:26 baboune do u know which SQL could generate the error?  I could try it within the postgres container
13:27 kaliya I don't know the exact query
13:28 aliemieshko_ after update 5.1 to 5.1.1 I get error: node-4 nova-oslo.messaging._drivers.impl_rabbit ERROR: AMQP server on 192.168.0.2:5673 is unreachable: Socket closed. Trying again in 30 seconds.
13:28 mattymo joined #fuel
13:28 aliemieshko_ what is wrong ?
13:29 aliemieshko_ all nodes
13:29 baboune kaliya: the public.notifications table has no such index
13:30 adanin joined #fuel
13:31 baboune kaliya: check https://dl.dropboxusercontent.com/u/18136096/psql-notifications.log
13:36 baboune kaliya: actually there does not seem to have such an index in postgres at all... u sure keystone uses the nailgun Postgres DB?
13:38 baboune is this correct?
13:44 baboune kaliya: Should I reset to my previous VM snapshot? Or do you want to investigate further?
13:51 e0ne joined #fuel
14:00 kaliya baboune: please revert the snapshot... we're investigating why the patch fills the disk again...
14:00 baboune kaliya: OK. Thx for helping out
14:03 baboune kaliya: snaphot revert worked!
14:03 kaliya baboune: :)
14:03 kaliya baboune: has the reverted snapshot already the patch applied?
14:03 kaliya please monitor the disk space and eventually tell us
14:04 emagana joined #fuel
14:05 baboune kaliya: will do
14:14 sovsianikov joined #fuel
14:17 jetole joined #fuel
14:19 jetole Good morning. I'm using neutron w/ vlan. I need to add additional networks, specific to certain VLAN's where there is no NAT i.e. the fixed IP can be reached from the router outside of the openstack cluster but I am having a hard time determining how to set this up. Could anyone please help me with this?
14:27 baboune kaliya: yes the reverted snapshot has the patch applied
14:39 kaliya baboune: please look also https://bugs.launchpad.net/fuel/+bug/1371757
14:49 evg jetole: Hello, using snat/dnat seem to be the only way for neutron. https://answers.launchpad.net/neutron/+question/222494
14:50 evg jetole: please send a note if you find something on the subject
14:53 emagana joined #fuel
14:54 jetole evg: I was reading this and it looks like a solution but I don't know well enough to say for sure. Can you provide any input? https://developer.rackspace.com/blog/neutron-networking-vlan-provider-networks/
14:57 baboune kaliya: looks very similar... I dont know how the other fix works but my disk space is relatively small ~50G for /var, and after a "normal" install, already 35% is occupied. this leaves only 33G for rotation
14:59 baboune kaliya: the main difference is that in my case it breaks simply because /var gets full w/o adding any additional nodes
15:03 baboune Added a comment upon https://bugs.launchpad.net/fuel/+bug/1371757
15:03 baboune but generally this is a poor solution to add a warning
15:04 baboune it is the second time that I get a failure following a disk /var getting full, and a warning would make 0 difference.  IMO log rotation should be based on the size of available disk.
15:06 vtzan_ joined #fuel
15:07 aliemieshko_ after update 5.1 -> 5.11 I got from rabbitmq error: AMQPLAIN login refused: user 'nova' - invalid credentials" . Log:   http://paste.openstack.org/show/137485/
15:17 baboune should there be a different bug report on /var and failing/corrupted docker containers as a result? It seems to me that log rotation is not sufficient. at least not if there is any chance of /var getting full.  A more permanent solution would be to allocate different space for logging and containers, so that one can not corrupt the other?
15:19 kaliya baboune: thank you for having commented out. I will report to our devs.
15:24 blahRus joined #fuel
15:30 angdraug joined #fuel
15:31 teran joined #fuel
15:33 byrdnuge joined #fuel
15:47 emagana joined #fuel
15:49 ArminderS joined #fuel
15:58 evg baboune: I launched a bug here https://bugs.launchpad.net/fuel/+bug/1383741 (referred to your experience, bwt)
16:00 evg baboune: +1 different partition for /var/logs is the right idea
16:29 akurenyshev joined #fuel
16:33 emagana joined #fuel
16:37 monester_laptop joined #fuel
16:41 emagana joined #fuel
16:59 mattgriffin joined #fuel
17:15 emagana joined #fuel
17:18 jetole I am using quantum/vlan as the default for tenant networking. Is there a way for me to create projects which do not use quantum/vlan. Quantum/vlan supports the majority or projects I have but I have special case projects which I need to set up NAT-less networking to the router
17:19 jetole s/quantum/neutron/g
17:25 rmoe joined #fuel
17:35 adanin joined #fuel
17:38 emagana joined #fuel
18:02 emagana joined #fuel
18:04 e0ne joined #fuel
18:06 jobewan joined #fuel
18:10 kupo24z What is the latest version os oslo messenger for 5.1.x?
18:10 kupo24z & is there an updated package I can install somewhere?
18:29 teran joined #fuel
18:32 monester_laptop joined #fuel
18:42 angdraug kupo24z: http://mirror.fuel-infra.org/fwm/5.1.1/
18:42 angdraug currently at 1.3.1, we have a 1.4.1 under review
18:43 angdraug https://bugs.launchpad.net/mos/+bug/1393505
18:43 emagana joined #fuel
19:05 xarses joined #fuel
19:06 emagana_ joined #fuel
19:09 emagana joined #fuel
19:13 coryc joined #fuel
19:41 emagana joined #fuel
19:45 e0ne joined #fuel
20:09 HeOS joined #fuel
20:24 agordeev joined #fuel
20:25 emagana joined #fuel
20:40 teran joined #fuel
20:51 angdraug joined #fuel
21:12 mbradfo74 joined #fuel
21:24 artem_panchenko left #fuel
21:31 artem_panchenko joined #fuel
21:34 Rajbir joined #fuel
21:34 xarses kupo24z: what version is this from? re: http://pastebin.mozilla.org/7476357
21:39 kupo24z 1s
21:39 kupo24z http://pastebin.mozilla.org/7520273
21:40 kupo24z xarses: ^
21:44 monester_laptop joined #fuel
21:46 adanin joined #fuel
21:49 emagana joined #fuel
21:54 emagana joined #fuel
22:04 mattgriffin joined #fuel
22:04 e0ne joined #fuel
22:07 emagana joined #fuel
22:09 emagana joined #fuel
22:18 kupo24z angdraug: what is the upgrade procedure when that oslo messenger package gets backported?
22:19 angdraug install updated package on all nodes, restart all openstack services
22:19 kupo24z angdraug: is there an order to which services should be restarted?
22:24 emagana joined #fuel
22:25 emagana joined #fuel
22:28 angdraug not particularly
22:28 angdraug controllers before computes, obviously
22:28 angdraug although not sure if it's that important
22:47 adanin joined #fuel
23:15 adanin joined #fuel
23:16 belph joined #fuel
23:17 belph Could I get some help with some Sahara errors?
23:17 belph I try to deploy a cluster and get an Error status
23:17 belph With no other message
23:18 belph HEY
23:20 emagana joined #fuel
23:21 emagana joined #fuel
23:21 xarses belph please use a paste service like http://paste.openstack.org to show us the error messages in the logs
23:21 xarses people that are around will attempt to help if they can
23:45 aleksandr_null joined #fuel

| Channels | #fuel index | Today | | Search | Google Search | Plain-Text | summary