Perl 6 - the future is here, just unevenly distributed

IRC log for #fuel-dev, 2014-04-29

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:37 rmoe joined #fuel-dev
01:45 xarses joined #fuel-dev
02:42 IlyaE joined #fuel-dev
03:37 warpc joined #fuel-dev
03:40 BillTheKat joined #fuel-dev
03:46 IlyaE joined #fuel-dev
04:25 Ch00k joined #fuel-dev
04:54 vkozhukalov joined #fuel-dev
05:04 IlyaE joined #fuel-dev
05:12 mattymo|home joined #fuel-dev
05:54 IlyaE joined #fuel-dev
06:11 artem_panchenko2 joined #fuel-dev
06:23 al_ex joined #fuel-dev
06:25 sambork joined #fuel-dev
06:25 alex_didenko joined #fuel-dev
06:28 bookwar fuel-ci doesn't work because of changed gerrit keys, fixing right now
06:28 crshman joined #fuel-dev
06:38 teran joined #fuel-dev
06:41 sambork Can I ask for an review https://review.openstack.org/#/c/89791/ ?
06:42 e0ne joined #fuel-dev
06:49 bookwar Fuel CI is alive and fully operational, and i've retriggered some failed jobs from yesterday
06:49 bookwar btw we still use iso 147 for ci, not the docker one
06:50 morale_ joined #fuel-dev
06:50 morale joined #fuel-dev
07:05 rzhnichkov_ joined #fuel-dev
07:08 artem_panchenko joined #fuel-dev
07:10 akislitsky mihgen, I guess we need to set priority of bug https://bugs.launchpad.net/fuel/+bug/1313797 to Critical
07:17 mrasskazov joined #fuel-dev
07:18 fandi joined #fuel-dev
07:19 mihgen akislitsky: is it on master?
07:19 akislitsky yes
07:20 mihgen akislitsky: set to critical. please add info from /api/version to the bug
07:21 akislitsky mihgen, done
07:29 Ch00k joined #fuel-dev
07:32 morale_ joined #fuel-dev
07:32 morale joined #fuel-dev
07:40 vkozhukalov joined #fuel-dev
07:57 mihgen mattymo: did you see  https://bugs.launchpad.net/fuel/+bug/1313797  ?
08:05 EeeBotu New bug #1314071 in fuel: "update diagnostic snapshot to get cobbler data via xmlrpc" [Undecided, New] https://bugs.launchpad.net/bugs/1314071
08:05 EeeBotu New bug #1314065 in fuel: "provisioning fails, controller vms hangs and not respond" [Undecided, New] https://bugs.launchpad.net/bugs/1314065
08:05 EeeBotu New bug #1314063 in fuel: "fuel cli snapshot doesn't work under docker" [Undecided, New] https://bugs.launchpad.net/bugs/1314063
08:11 teran joined #fuel-dev
08:39 mattymo|home joined #fuel-dev
08:41 anotchenko joined #fuel-dev
08:43 b-zone joined #fuel-dev
08:45 b-zone left #fuel-dev
08:53 vrovachev joined #fuel-dev
09:00 evgeniyl mihgen: 155 iso installation fails because there was missed busybox container, mattymo is working on it.
09:00 anotchenko joined #fuel-dev
09:01 teran joined #fuel-dev
09:02 mattymo|home it worked when I deployed because I had internet access and it pulled busybox from docker's repo
09:02 mattymo|home we can solve this going forward in 2 ways - 1 is nightly fuel-images.tar.lrz that gets uploaded after smoke test completes
09:03 mattymo|home and 2 is we need to merge my fix for building iso from scratch: https://review.openstack.org/#/c/90939/
09:05 BillTheKat joined #fuel-dev
09:07 EeeBotu New bug #1314085 in fuel: "HA mode, Murano -> deployment failed." [Undecided, New] https://bugs.launchpad.net/bugs/1314085
09:10 teran joined #fuel-dev
09:19 sambork joined #fuel-dev
09:22 mirrorbox will plugins be also supported in fuelclient?
09:23 mirrorbox I want to add an additional node attribute and wondering if I should extend cli/actions/node.py or implement everything as a separate action
09:24 mirrorbox (which would generally look very similar to node.py still)
09:24 anotchenko mirrorbox: It depends
09:25 mirrorbox depends on what?
09:25 anotchenko Is it some patching just for you, or you planning to try to merge it someday?
09:25 mirrorbox plan to merge
09:25 anotchenko And what is the attribute you want to add?
09:26 mirrorbox raid configuration for the node
09:27 anotchenko mirrorbox: as original dev on CLI, I intended all attributes to be in node action, but it possible to refactor attributes as sub-sub-parsers some day
09:28 anotchenko I recomend adding code to cli/actions/node.py
09:28 mirrorbox good, thanks
09:28 mirrorbox anyway, if plugins will be supported by the nailgun, I guess fuelclient should support plugins as well :)
09:30 ikalnitsky mattymo: hi. could you please help me? i've met a strange issue. when i run a shotgun from mcollective containers -- all works fine. when i try to make snapshot via web ui -- the shotgun can't collect info from master node. the config is correct.   any ideas?
09:34 evgeniyl ikalnitsky: take a look at orchestrator's logs
09:34 evgeniyl ikalnitsky: does it run shotgun via mcollective?
09:35 ikalnitsky yes, it runs. and a shapshot is generated.
09:35 ikalnitsky but information is empty
09:35 ikalnitsky the dump log in this case show that during connection to master node the password was prompted. \
09:36 ikalnitsky i add mcollective's ssh key to master node, so when i run shotgun manually -- all generated is ok, without asking a password
09:36 mattymo|home ikalnitsky, are you trying to fix shotgun to run in docker?
09:36 ikalnitsky yes
09:37 mattymo|home I'm currently working on this task as well... so maybe we can work together, but let's not try to approach it in 2 different ways
09:37 mattymo|home the one ticket you took is nailgun sends the wrong db host ... that needs nailgun code fixes
09:37 ikalnitsky mattymo|home: i've fixed it
09:38 mattymo|home oh great
09:38 ikalnitsky do you need them? i can push it on review
09:38 mattymo|home please just propose that patch - we can get it merged quick
09:39 mattymo|home the parts we need ssh for: dmidecode, brctl, pvdisplay, vgdisplay, lvdisplay
09:39 mattymo|home cobbler should be rewritten to use xmlrpc
09:39 ikalnitsky mattymo|home: https://review.openstack.org/90957
09:40 ikalnitsky mattymo|home: the settings.yaml has incorrect settings for postgres, but it's overrides by nailgun during generating config for shotgun
09:44 mattymo|home ok as long as it works :)
09:45 mattymo|home ikalnitsky, for ssh key dilemma: we should copy /root/.ssh/id_rsa.pub to /root/.ssh/authorized_keys2 and then expose this key to mcollective container like we do for cobbler
09:45 mattymo|home I'll propose a patch for that side
09:46 mattymo|home shotgun can ssh to the ip of its gateway, what's in /etc/fuel/astute.yaml, or MASTER_IP from nailgun settings.yaml
09:46 mattymo|home all will be fine
09:49 ikalnitsky mattymo|home: i temporary add the key to master manually. so, let me descrive the situation i have one more time. I press on "Generate diagnostic snapshot" on web ui. Nailgun generate the config file and send it to astute. It saves it under /tmp/dump_config and send to mcollective a command to run shotgun -c /tmp/dump-config. The shotgun runs, but can't connect to master node to collect some logs and execute some commands.
09:50 ikalnitsky But when I run this command manually from mcollective's container, the snapshot is built succeccfully. What's going on? I can't figured out right now.
09:51 ikalnitsky here is generated config http://paste.openstack.org/show/77651/
09:52 mattymo|home ikalnitsky, imagine a docker container is like a virtual machine
09:52 mattymo|home it can't see processes or files outside of its contanier
09:52 mattymo|home sshing to master node is one option. sharing files via docker is another way that will suit some purposes
09:56 ikalnitsky mattymo|home: sorry, i can't understand what you're talking about. perhaps i misunderstand something. as far as i understand, astute send a command to run shotgun via rabbitmq. mcollective receive this command and run it inside the mcollective container. so what difference between running command by mcollective and manually running command inside mcollective container?
09:56 mattymo|home ikalnitsky, also avoid the word actualize...
09:57 dukov joined #fuel-dev
10:01 teran_ joined #fuel-dev
10:08 EeeBotu New bug #1314128 in fuel: "Nodes failed to reboot  on simple flat environment" [High, Confirmed] https://bugs.launchpad.net/bugs/1314128
10:27 e0ne joined #fuel-dev
10:32 mattymo|home ok all - off to the conference
10:33 anotchenko joined #fuel-dev
10:44 evgeniyl bookwar: hi, warpc added new patch https://review.openstack.org/#/c/87533 but fuel ci didn't start tests, how can we run tests or warpc should upload new patch?
10:46 bookwar it is in the queue
10:48 evgeniyl bookwar: oops, sorry
10:48 bookwar i am working on parallel test runs for this job right now, but it is not ready yet
11:00 teran joined #fuel-dev
11:07 e0ne joined #fuel-dev
11:09 e0ne joined #fuel-dev
11:10 EeeBotu New bug #1314147 in fuel: "Network verification failed: Queue.declare: (406) PRECONDITION_FAILED" [Undecided, New] https://bugs.launchpad.net/bugs/1314147
11:10 ikalnitsky mattymo|home: evgeniyl: so, as far as i understood mat's comments here https://review.openstack.org/#/c/90957/ , we have to collect log files from mcollective container, not from master node. right? from master node we need to get some /etc/ files and some command outputs, right?
11:11 e0ne joined #fuel-dev
11:13 evgeniyl ikalnitsky: this directory binded to mcollective container /var/log, but I don't see any problems to do it over ssh.
11:15 ikalnitsky evgeniyl: yes, it'll work in both cases. but what approach is better?
11:16 evgeniyl ikalnitsky: I think it will better to reduce amount of shell commands which we run inside of container.
11:28 ikalnitsky evgeniyl: ok, thanks. and one more question about postgres: where pg_dump should be called? inside mcollective container or outside any of containers?
11:29 e0ne joined #fuel-dev
11:30 evgeniyl ikalnitsky: it should be called inside of container https://github.com/stackforge/fuel-main/commit/ba2c9fec6f0d9585b575eae5b6d9d3b912819dd3 here postgresql command line utils were added as dependencies for shotgun.
11:42 morale_ joined #fuel-dev
11:42 morale joined #fuel-dev
11:45 bookwar WARN: Fuel CI doesn't comment on Gerrit right now, though it runs all the tests
11:52 al_ex joined #fuel-dev
12:11 EeeBotu New bug #1314170 in fuel: "Missing documentation: building ISO using Docker" [High, New] https://bugs.launchpad.net/bugs/1314170
12:11 EeeBotu New bug #1314165 in fuel: "Could not find resource 'Exec[ostf-init]' for relationship from 'Postgresql::Db[ostf]'" [Critical, New] https://bugs.launchpad.net/bugs/1314165
12:27 e0ne joined #fuel-dev
12:29 ikalnitsky evgeniyl: mattymo: ok, i've found why shotgun can't ssh to master when has been started by mcollectuve. the issue is that HOME env variable is incorrect in case of mcollective. don't you know, guys, where can I change it?
12:29 bookwar Fuel CI trigger fixed now, but there might be issues with already sent patchsets
12:29 akupko joined #fuel-dev
12:29 bookwar i am going to retrigger jobs, but if i forget some, send me the link
12:29 rzhnichkov_ joined #fuel-dev
12:31 evgeniyl ikalnitsky: can you use different directory for keys? We can pass it as parameter from nailgun.
12:32 evgeniyl ikalnitsky: or we can set right variable in orchestrator
12:39 Arminder- joined #fuel-dev
12:41 BillTheKat joined #fuel-dev
12:41 warpc joined #fuel-dev
12:42 ikalnitsky evgeniyl: right variable is settuped. if i run the shotgun manually, the HOME is '/root', but when it runs by mcollective the HOME is '/'. I guess it's incorrect, and should be fixed. Don't you think so? I don't like idea to pass directory as param from nailgun. Looks a bit hackish.
12:45 evgeniyl ikalnitsky: it will be hardcoded anyway, the question is where we need to hardcode it, we can hardcode it in orchestrator, and I think it's not the best way, we already have configurations which nailgun sends to orchestrator, but if it difficult to pass directory with keys, we can ask warpc to add hack for this with todo, it should not be hardcoded in
12:45 evgeniyl orchestrator anyway.
12:56 ikalnitsky evgeniyl: i have a quick look around fabric api. it looks like we can easily setup a key to use (with path to keyfile or key itself). so, it's not difficult to do. but i don't know.. i don't like this idea. maybe mcollective config has some option, for example "home_env" or something like that? or it's bad idea?
12:59 BillTheKat joined #fuel-dev
13:08 evgeniyl ikalnitsky: it's not technical issue, we can hardcode it in orchestrator, mcollective, but lets think from architecture point of view, shotgun is a command line interface which can retrieve some data from nodes, and it requires ssh keys, and it's work should not depend on some variables from env (it can, but only if exact path was not provided), and this
13:08 evgeniyl keys can be in any other directory, not only in /root they can be in /home/some_user or just /directory . Ssh keys path is part of config, we generate configs by nailgun, as result this path to keys should be in nailgun.
13:13 salmon_ joined #fuel-dev
13:14 EeeBotu New bug #1314190 in fuel: "ntpdate start takes 1 minute and fails on virtualbox" [Low, New] https://bugs.launchpad.net/bugs/1314190
13:15 ikalnitsky evgeniyl: ok, then. i'll do it right now.
13:17 vkramskikh guys, why do you cancel jobs on jenkins?
13:17 vkramskikh i've been waiting for a few hours for a jib start
13:17 vkramskikh and now i see that the job is cancelled
13:17 vkramskikh wtf?
13:18 vkramskikh https://review.openstack.org/#/c/90741/ this request for example
13:18 vkramskikh i saw a scheduled job for it, but cannot see it now
13:18 vkramskikh stop doing it please
13:19 vkramskikh i know there is a queue, but you are not alone who is waiting for the tests
13:21 e0ne joined #fuel-dev
13:21 bookwar vkramskikh: Fuel CI needed to be restarted because of Gerrit Trigger update. Overwise it couldn't post comments on Gerrit
13:22 vkramskikh ok, what should i do now to run the tests against that change request?
13:24 bookwar retriggered now, i am going through the whole list but slowly, thus if you have some direct links - send them to me
13:33 Tatyanka_Leontov joined #fuel-dev
13:59 Ch00k joined #fuel-dev
14:15 EeeBotu New bug #1314210 in fuel: "Add Openstack logs in UI (for all cases and services)" [Undecided, Triaged] https://bugs.launchpad.net/bugs/1314210
14:15 EeeBotu New bug #1314200 in fuel: "MongoDB keyfile should be generated" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1314200
14:15 EeeBotu New bug #1314196 in fuel: "glance doesn't send notifications to ceilometer" [Undecided, New] https://bugs.launchpad.net/bugs/1314196
14:15 EeeBotu New bug #1314194 in fuel: "Missing documentation: repo for docker (4.4. Building the Fuel ISO)" [Medium, Confirmed] https://bugs.launchpad.net/bugs/1314194
14:45 anotchenko joined #fuel-dev
14:54 jobewan joined #fuel-dev
15:01 IlyaE joined #fuel-dev
15:04 morale_ joined #fuel-dev
15:04 morale joined #fuel-dev
15:05 akupko joined #fuel-dev
15:05 e0ne joined #fuel-dev
15:08 rzhnichkov_ joined #fuel-dev
15:10 ikalnitsky evgeniyl: ok, i've implemented custom ssh key for shotgun, but the issue is still there. it looks like fabric internally is used somewhere $HOME.
15:18 EeeBotu New bug #1314227 in fuel: "Slave nodes can not install Ubuntu: Debootsrap warning: passwd 3.5.24_64" [Critical, Confirmed] https://bugs.launchpad.net/bugs/1314227
15:18 EeeBotu New bug #1314224 in fuel: "Wrong data appears in astude.yaml after fuel menu was called" [High, New] https://bugs.launchpad.net/bugs/1314224
15:22 morale_ hi all! can anyone assit with the gerrit? I have an issue after running gir review - >  Problem running 'git remote update gerrit'
15:22 morale_ Fetching gerrit
15:22 morale_ Permission denied (publickey).
15:22 morale_ fatal: Could not read from remote repository.
15:22 morale_ Please make sure you have the correct access rights
15:22 morale_ and the repository exists.
15:22 morale_ error: Could not fetch gerrit
15:22 morale_ tried resetting ssh keys but no luck
15:25 aglarendil try to check if your ssh public key is uploaded to gerrit
15:26 aglarendil firstly by checking your profile settings
15:26 aglarendil on review.openstack.org
15:27 xarses joined #fuel-dev
15:27 evgeniyl ikalnitsky: could you please share you code somewhere? For example you can fork fuel-web on github and push you changes there.
15:31 rmoe joined #fuel-dev
15:33 xdeller joined #fuel-dev
15:35 ikalnitsky evgeniyl: sure, i can. btw, i dive in this problem. something wrong with key validation inside paramiko, when HOME is "/".
15:35 rmoe I'm working on some neutron issues and I noticed that the l3-agent was creating all of its files as 751, I checked the umask inside the OCF script and it is 026 (l3-agent has the same umask obviously)
15:36 rmoe I have no clue why the umask is 026
15:36 rmoe anybody have an idea?
15:36 aglarendil ok. what is your concern? this should not break things. obviously it looks like a typo
15:36 rmoe this is centos btw, I haven't tested on ubuntu yet
15:37 ikalnitsky evgeniyl:: here the branch -- https://github.com/ikalnitsky/fuel-web/tree/bug/1312620
15:39 rmoe aglarendil: the problem is that a 026 umask causes files to be created 751, the l3-agent creates some files as root and then tries to read them later as the neutron user which will obiovulsy fail
15:39 rmoe I don't see anywhere in any ocf script where the umask is being set, so I don't understand where 026 is coming from
15:40 aglarendil hmmm
15:40 aglarendil wtf
15:41 rmoe also, this is on 4.1.1
15:43 evgeniyl ikalnitsky: strange, it should work, try to print 'keys' variable here https://github.com/fabric/fabric/blob/4ad127be0a42fc40512ab9138e465524c0a39d36/fabric/network.py#L218
15:44 evgeniyl ikalnitsky: I think you need to use private key, not public.
15:44 aglarendil rmoe: and what is the group of created files?
15:44 ikalnitsky evgeniyl: i've printed. that's ok. but i just figured out that paramiko failed on checking private key
15:44 aglarendil rmoe: if it is neutron, than everything is ok
15:44 rmoe aglarendil: root:root
15:44 aglarendil wow
15:44 ikalnitsky evgeniyl: so yes, you're right. will test it now.
15:44 rmoe which would be fine if the umask were correct
15:45 aglarendil actually, we should create files with correct group
15:45 aglarendil or set SGID bit on the directories
15:45 rmoe the neutron code does it that way
15:45 aglarendil by the way, which files does it create ?
15:45 rmoe we shouldn't have a umask of 026 for our processes
15:45 rmoe the pid file for the metadata proxy
15:45 rmoe it creates it as root and tries to read it later as neutron
15:46 aglarendil interesting
15:46 aglarendil it looks like it is 0755 in manifests
15:47 rmoe I checked the ns_haproxy script and it also has a umask of 026
15:47 rmoe so it appears that all scripts have that umask
15:47 aglarendil it is f**ing awesome
15:48 rmoe what actually calls these scripts?
15:48 aglarendil puppet creates them
15:48 rmoe what actually executes them?
15:49 aglarendil pacemaker runs them
15:49 aglarendil just grep for 751 and change it to 755 in manifests
15:49 rmoe how will that fix anything?
15:50 ikalnitsky evgeniyl: ok, it works pretty fine now. :) thanks
15:51 aglarendil these files are created by puppet
15:51 aglarendil and it looks like we had typo there
15:52 rmoe which files?
15:52 xenolog rmoe, that files you mean?
15:53 rmoe when a router is created neutron spawns an ns-metadata-proxy process, that pidilfe has 751 permissions because the l3-agent proceess has a umask of 026
15:53 rmoe these pidfiles are in /var/lib/neutron/external/pids
15:54 rmoe the reason the l3-agent has a umask of 026 is because the ocf script which launches it has a umask of 026
15:54 rmoe but I don't understand why the umask is 026 for ocf script
15:55 rmoe that's what I'm trying to figure out
15:55 evgeniyl ikalnitsky: cool, also I would suggest to test you with some deployed nodes, you can create cluster with single controller, wait when deployment is done, and then run diagnostic snapshot generation, and make sure that there are logs from this controller in the snapshot.
15:56 evgeniyl ikalnitsky: I'm afraid there could be some issues with keys too.
15:58 xenolog O! yep!
15:59 xenolog as I known, we don't setup umask inside ocf scripts.
15:59 rmoe exactly
16:00 rmoe so it must be inheriting the umask from somewhere right?
16:02 ikalnitsky evgeniyl: yes, sure. i'll test it. but one moment here -- the ssh-key should be in the authorized_keys file on the slaves. it will be done?
16:03 evgeniyl ikalnitsky: yeah, it worked before, but lets check it :)
16:03 xenolog O!
16:03 xenolog When any ocf script try to create network namespace,
16:03 xenolog one use this netmask:
16:03 xenolog [root@node-1 neutron]# ls -al /var/run/ | grep netns
16:03 xenolog drwxr-x--x   2 root      root      4096 Apr 29 14:51 netns
16:04 rmoe yeah, as far as i can tell all ocf scripts have a 026 umask
16:06 xenolog but grep inside /etc/ can't found this value
16:06 rmoe I can't either
16:06 rmoe I have no idea how it gets set to 026
16:06 rmoe but it's definitely a problem for the l3-agent
16:07 rmoe and maybe other resources, 026 doesn't seem like a very useful mask to me
16:07 rmoe one thing I noticed is that 026 == 22 decimal, so maybe the umask of 022 is being treated as decimal somewhere?
16:08 rmoe but I can't find anything that looks like that
16:09 xenolog rmoe: we already had merged death any network-based processes inside net.namespace when agents was stopped
16:10 xenolog but it's interesting…
16:10 rmoe yes, I understand that, but this should still be fixed
16:10 rmoe because 026 is almost certainly not what we want
16:11 mattymo|home Tatyanka_Leontov, bogdando aglarendil I'm back if you have any burning issues
16:11 xenolog we should found it before.
16:11 mattymo|home I'm going to test bogdando and alex_didenko's patches
16:12 enikanorov__ joined #fuel-dev
16:12 enikanorov__ joined #fuel-dev
16:12 salmon_ joined #fuel-dev
16:16 sanek joined #fuel-dev
16:33 xenolog rmoe: I check now. inside ocf script umask is realy 0026,  but ip utility, when creates /var/run/netns , creates it with 0755, but directory creates with 'drwxr-x--x'
16:34 akupko joined #fuel-dev
16:39 angdraug joined #fuel-dev
16:39 dshulyak_ joined #fuel-dev
16:58 vkozhukalov joined #fuel-dev
17:00 Ch00k joined #fuel-dev
17:08 MiroslavAnashkin If we configure Ceph as backend for all with RadosGW - should we add role "swiftoperator" to keystone and assign it with a user to allow this user to manage Object Store containers?
17:13 teran joined #fuel-dev
17:16 teran joined #fuel-dev
17:21 teran_ joined #fuel-dev
17:21 Ch00k joined #fuel-dev
17:22 teran_ joined #fuel-dev
17:35 xenolog rmoe: It's a pacemaker. hardcored inside C-code
17:38 rmoe xenolog: where in the code?
17:38 rmoe I looked a little bit yesterday and didn't find it
17:41 dshulyak_ joined #fuel-dev
17:48 rmoe looks like it's in a few places
17:48 rmoe S_IWGRP | S_IWOTH | S_IROTH = 026 right?
17:56 xenolog yep
17:56 rmoe well that's terrible
18:06 jaypipes joined #fuel-dev
18:20 teran joined #fuel-dev
18:27 mattymo|home aglarendil, you still here?
18:32 dshulyak_ joined #fuel-dev
19:06 fandi joined #fuel-dev
19:31 mrasskazov joined #fuel-dev
19:39 IlyaE joined #fuel-dev
20:30 IlyaE joined #fuel-dev
20:42 salmon_ joined #fuel-dev
20:57 e0ne joined #fuel-dev
21:01 aglarendil hey, Matt. you can sleep well. smoke test passed
21:12 salmon_ joined #fuel-dev
21:40 teran joined #fuel-dev
22:37 MiroslavAnashki- joined #fuel-dev
22:38 crshman joined #fuel-dev
22:42 teran_ joined #fuel-dev
22:46 e0ne joined #fuel-dev
22:55 xenolog_ joined #fuel-dev
23:06 e0ne joined #fuel-dev
23:39 mrasskazov joined #fuel-dev
23:50 rmoe joined #fuel-dev

| Channels | #fuel-dev index | Today | | Search | Google Search | Plain-Text | summary