Perl 6 - the future is here, just unevenly distributed

IRC log for #salt, 2014-01-14

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:06 phillip_ joined #salt
00:07 clearclaw left #salt
00:07 psyl0n joined #salt
00:08 clearclaw joined #salt
00:09 jacksontj joined #salt
00:11 phillip_ joined #salt
00:13 jpeach to answer myself, this works perfectly; just need to remember to call saltutil.sync_all when I'm testing it :)
00:17 elithrar joined #salt
00:20 jpeach left #salt
00:22 taion809 joined #salt
00:23 wking joined #salt
00:29 adizam joined #salt
00:31 adizam Is this a good place to ask a question? Using salt to provision vagrant boxes in development.. wondering the best approach to picking package versions for install (if I go with default nodejs, it installs 0.6.. I need to specify 0.10). Is it possible to do this from a sls file? (nodejs@0.10.24 -> pkg.installed)
00:31 troyready joined #salt
00:31 Nexpro joined #salt
00:34 __number5__ adizam: yes you can, see the version parameter http://docs.saltstack.com/ref/states/all/salt.states.pkg.html#salt.states.pkg.installed
00:35 adizam __number5__: Ty!
00:36 ndrei joined #salt
00:36 yomilk joined #salt
00:36 psyl0n joined #salt
00:45 jimallman joined #salt
00:47 MTecknology dave_den: sorry, ran off
00:48 MTecknology dave_den: I didn't know {% set ... %} THANKS!!!
00:48 austin987 joined #salt
00:53 MTecknology dave_den++
00:53 wking state.sls has 'exclude' to blacklist IDs.  Is there a way to whitelist IDs instead?  I'd like to test a new ID without running the whole sls state.
00:56 gnu-man joined #salt
00:59 clearclaw TypeError: unsupported operand type(s) for -: 'StrictUndefined' and 'StrictUndefined'
00:59 clearclaw TypeError: unsupported operand type(s) for -: 'StrictUndefined' and 'StrictUndefined'
00:59 clearclaw I'm getting that out of current Salt.
00:59 clearclaw http://pastebin.com/svfGbxYA  The full traceback
01:00 clearclaw As we're heading into a push, a quick pointer would be REAL helpful
01:00 KyleG unsupported operand type(s) for -: 'StrictUndefined' and 'StrictUndefined'
01:00 KyleG yeah
01:00 KyleG Paste your zabbix-agent sls file
01:01 KyleG not here.
01:01 KyleG Into gist.github.com
01:01 KyleG or that pastebin
01:01 clearclaw The SLS: http://pastebin.com/VJmwg4Af
01:01 clearclaw Hurm, truncated line there.  Fixing.
01:02 KyleG hrm
01:03 clearclaw Fixed: http://pastebin.com/3bZ4v9s1
01:04 KyleG I don't use grains or context, but I'm betting it's something there…have you tried removing the zabbix_register part and running it to see if that's the part that's broken
01:04 KyleG process of eilimination
01:04 clearclaw Not yet.
01:04 KyleG elimination*
01:04 clearclaw If I had a guess, I'd say the mine business...
01:04 clearclaw Though that was working...
01:05 NotreDev joined #salt
01:07 troyready joined #salt
01:07 KyleG Sorry I couldn't be more help...
01:09 clearclaw Log from the minion: http://pastebin.com/QHka3iw9
01:10 KyleG So the line is
01:10 KyleG host_ip: {
01:10 phillip_ left #salt
01:14 robawt aloha cool peeps
01:14 bhosmer joined #salt
01:14 robawt anyone work with postgresql and salt and not suffer enough PTSD to talk about it?
01:15 robawt my question stems from UTF-8 encoding: should I still explicitly create databases with UTF-8 encoding even though the DB's postgresql.conf will be set for default to utf-8
01:19 psyl0n joined #salt
01:20 psyl0n joined #salt
01:20 __number5__ PTSD?
01:21 KyleG Post Traumatic Stress Disorder
01:21 KyleG It's very real, and it sucks very much.
01:23 __number5__ I don't set any encoding in sls with postgresql it works fine
01:25 dpn` yeah it works fine until you get something that isn't ascii
01:25 dpn` robawt, set the encoding
01:28 __number5__ all my db are unicode/utf-8
01:28 dpn` look at lc_type encoding and lc_locale params to postgresql.present
01:28 dpn` __number5__, cool - then you set up the DB with the right encoding at install
01:28 dpn` ubuntu does not do this by default
01:29 dpn` it defaults to latin1 iirc
01:29 __number5__ I guess my OS(ubuntu) has same encoding as postgres in this case
01:30 yomilk joined #salt
01:30 dpn` maybe you have a more recent version than 12.04
01:31 __number5__ 12.04 LTS, or 12.04.3
01:36 honestly joined #salt
01:41 cachedout joined #salt
01:43 Ryan_Lane2 joined #salt
01:43 dpn` __number5__, what postgres version?
01:44 deepakmdass joined #salt
01:44 __number5__ 9.1
01:44 dpn` or rather what have you installed that is not out of the default apt repos
01:44 dpn` did you change any configuration in postgresql.conf?
01:44 __number5__ yes, I've updated postgresql.conf
01:45 dpn` what did you set for utf8 stuff?
01:49 __number5__ everything for lc_* is set to 'en_US.UTF-8'
01:59 Ryan_Lane2 joined #salt
02:03 higgs001 joined #salt
02:05 drawsmcgraw joined #salt
02:06 ConceitedCode joined #salt
02:10 elfixit joined #salt
02:10 Ryan_Lane2 joined #salt
02:10 deepakmdass joined #salt
02:13 Ryan_Lane2 joined #salt
02:13 ajw0100 joined #salt
02:30 jfzhu_us joined #salt
02:30 njs126 joined #salt
02:36 yomilk joined #salt
02:37 Ryan_Lane2 joined #salt
02:45 xl1 joined #salt
13:41 ilbot3 joined #salt
13:41 topic for #salt is now Welcome to #salt - SaltConf Jan 28-30, 2014! http://saltconf.com (reg deadline January 3) | 0.17.4 is the latest | Please be patient when asking questions as we are volunteers and may not have immediate answers - Channel logs are available at http://irclog.perlgeek.de/salt/
13:42 __number5__ manage.boostrap is only available in latest version (aka 2014.1.0rc1)
13:46 Katafalk_ joined #salt
13:49 juasiepo joined #salt
13:52 bhosmer joined #salt
13:55 ajw0100 joined #salt
13:58 backjlack joined #salt
14:09 juicer2 joined #salt
14:09 brianhicks joined #salt
14:13 Gifflen joined #salt
14:14 elithrar joined #salt
14:14 deepakmdass joined #salt
14:15 Whissi joined #salt
14:18 ekristen is there any way to use mine.get to query on two grains at the same time to get the info I need?
14:19 racooper joined #salt
14:21 BogdanR Hello, for ssh_auth I see that I for installing keys I can specify them as present and I also have an option to specify them as absent when I want to revoke an user.
14:21 BogdanR My question is how do I make sure that manually added keys are not kept in the file?
14:22 timoguin joined #salt
14:22 ekristen sroegner: morning, that query you gave me the other day for mine.get on the command line, I can’t seem to find it do you still have it?
14:24 godber1 joined #salt
14:25 iri- Where does default pillar configuration come from? For example, with the saltstack-formulas/salt-formula? If I run the pillar.items, I see a load of configuration which I haven't specified
14:29 iri- hm. It looks like the defaults I see from pillar.items are the things specified inside the template here:  https://github.com/saltstack-formulas/salt-formula/blob/master/salt/files/master
14:30 iri- How on earth does that work?
14:31 timoguin iri-, master configuration gets loaded into pillar by default
14:31 whiskybar joined #salt
14:32 timoguin even if you weren't using that formula you'd see that pillar data
14:32 ekristen one of my minions is saying it is waiting for the master to accept the key
14:32 ekristen but I have no unaccepted keys
14:33 psyl0n joined #salt
14:33 NotreDev joined #salt
14:34 iri- timoguin: ahh! how confusing
14:35 logix812 joined #salt
14:35 sroegner ekristen: hi
14:36 ekristen sroegner: hi
14:36 sroegner ekristen: salt-call mine.get 'roles:hadoop_master' network.interfaces grain
14:37 sroegner http://docs.saltstack.com/ref/modules/all/salt.modules.mine.html#salt.modules.mine.get
14:37 ekristen hrm, wtf, my salt-call disappeared
14:37 sylvinus joined #salt
14:38 sroegner minion installed?
14:38 ekristen ah, this master doesn’t have the minion installed, I need to fix that
14:38 BogdanR How can salt install itself on a remote machine?
14:39 BogdanR https://github.com/saltstack-formulas/salt-formula or this is something else?
14:39 ekristen do you know if you can do something like mine.get ‘roles:first_role and roles:second_role’ network.interfaces grain
14:42 sroegner ekristen: compound matching appears to work
14:43 sroegner salt-call mine.get 'G@roles:hadoop_master and G@roles:hadoop_slave' network.interfaces compound
14:43 ekristen ah, you have to add the G@ — thanks, still getting use to this ;)
14:45 krak3n` joined #salt
14:46 mgw joined #salt
14:47 erjohnso joined #salt
14:48 sylvinus_ joined #salt
14:49 sylvinus joined #salt
14:50 sylvinus_ joined #salt
14:50 tyler-baker joined #salt
14:52 sylvinus joined #salt
14:54 BogdanR So, can the salt master install it's own minion on a different machine?
14:56 quickdry21 joined #salt
14:57 timoguin BogdanR, I'm not sure what you mean. I don't understand that use case.
14:57 MTecknology Is it pillar.get('pillar_item', 'default_value') ?
14:58 BogdanR In the salt-formulas there is a formula "salt-formula" which sais that salt can install itself
14:58 BogdanR timoguin: I wanted to know if somehow I can command the master to install the minion on a remote machine
14:58 timoguin BogdanR, not that I know of, no.
14:58 timoguin I have master and minion on the same machine for my masters.
14:59 timoguin MTecknology, yea that's right
14:59 MTecknology thanks
15:00 timoguin BogdanR, oh you mean to just have the salt-master provision a new minion machine?
15:00 imaginarysteve joined #salt
15:01 wolfpackmars2 joined #salt
15:01 Teknix joined #salt
15:02 jfxura joined #salt
15:02 timoguin BogdanR, if that's what you mean, then yes. you can use that formula to manage the salt-minion package and processes on other machines
15:09 luimemee joined #salt
15:09 luimemee hello is there an python api for salt-ssh ?
15:10 iri- when I run salt-call, is everything run from this process, or does salt-call communicate with other processes to make it do its bidding?
15:10 rlarkin joined #salt
15:11 jimallman joined #salt
15:13 vejdmn joined #salt
15:15 AdamSewell joined #salt
15:16 tomasfejfar BogdanR: Yes as of v.2014.1.0rc1 there is salt-run manage.bootstrap host="hostname" command, but no idea how it works exactly. I'm using latest stable.
15:19 timoguin tomasfejfar, oh cool!
15:19 juicer2 joined #salt
15:19 timoguin https://github.com/saltstack/salt/blob/develop/salt/runners/manage.py#L250
15:19 timoguin it let's you re-run the bootstrap script
15:19 JasonSwindle joined #salt
15:19 timoguin that's good to know about
15:20 tomasfejfar But you need to use dev-version, which is cool but not prefferable in production ;) Do you know when this hits the stable?
15:21 timoguin I know they're aiming to release 2014.1 in time SaltConf later this month
15:23 aleszoulek joined #salt
15:29 Linz joined #salt
15:30 teskew joined #salt
15:31 abe_music joined #salt
15:35 sylvinus joined #salt
15:39 ajw0100 joined #salt
15:39 achileuss left #salt
15:40 bhosmer joined #salt
15:40 tomasfejfar timoguin: that's great, I'm just starting to deploy some servers so it may make sense to use dev version... :)
15:41 kermit joined #salt
15:42 NotreDev joined #salt
15:42 alunduil joined #salt
15:44 patrek joined #salt
15:45 _rob_____ joined #salt
15:47 iri- what's the best way to import a formula into my /srv/salt repo?
15:50 Marion_ joined #salt
15:50 timoguin iri-, I just clone the git repo and copy them over.
15:50 timoguin But they're designed so you can just add them to your gitfs_remotes
15:50 timoguin and salt will pull them in
15:52 iri- timoguin: but then I lose the git history and everything :(
15:52 iri- and knowing what version I imported it, where I imported it from, etc.
15:52 gnu-man joined #salt
15:52 iri- timoguin: I'd quite like to use a git subtree merge, but the path would then be at /srv/salt/formulas/ssh-formula/ssh/ or something. Is there a way to make salt cope with that?
15:54 timoguin iri-, I'm not sure about subtree in Git, as I don't use it. There was some talk recently on the mailing list about it.
15:54 timoguin here: https://groups.google.com/forum/#!searchin/salt-users/salt$20formulas/salt-users/zx1VHnGx2lI/_cs_6Ta3z2EJ
15:54 timoguin I don't use the standard /srv/salt system. All my states and pillar are in gitfs
15:55 iri- timoguin: thanks for the link
15:55 iri- timoguin: but then you have the problem that your states might move under you and break, if you directly use a gitfs from somewhere you can't trust.
15:56 aleszoulek joined #salt
15:56 iri- timoguin: and many repositories to keep track of, potentially. I just want the one, really, but with lineage information so I can see where it came from
15:57 timoguin yea they discussed a few possible ways to cope with that in the thread i posted
15:57 timoguin I use my own repos and don't directly link to Github
15:57 timoguin as I don't want shit changing without me knowing
15:57 iri- so, forks, essentially
15:57 timoguin it'd be nice to clone just one repo and get all the formulas
15:57 timoguin yea
15:58 iri- Yeah, that's what I'm going for. And I guess you'd still have "forks", but they would just live inside the one git repo
15:58 iri- subtree merge, if you're not familiar, allows you to merge part of one git repository into another, in such a way as to preserve the history and allow upstreaming, whilst having the contents of the new repository living at a different path
16:01 Veticus joined #salt
16:02 teskew joined #salt
16:07 analogbyte does anybody know how to restrict a custom module from running on minions? I am trying to make a module of mine master-only, and I can't find something like that in salt.utils
16:08 WarheadsSE joined #salt
16:09 jcockhren analogbyte: that'll be a runner
16:10 ekristen how do I see what states are resolving to a minion?
16:10 timoguin ekristen, do you mean seeing what states will be applied to a highstate?
16:11 ekristen yes
16:11 salt_whitebelt joined #salt
16:11 timoguin you can add test=True to a highstate run
16:11 timoguin i'm not sure how to just get a list of the states though
16:11 ekristen just like salt ‘*’ state.highstate test=True
16:11 timoguin yes
16:11 ekristen ok
16:11 analogbyte jcockhren: mh yeah, that actually makes a lot of sense... I guess it's trivial to make the module a runner, thanks for the push in the right direction ;)
16:11 ekristen timoguin: another question
16:12 higgs001 joined #salt
16:12 ekristen timoguin: how can I make a minion run highstate when it first joins a master? using reactors?
16:12 cachedout joined #salt
16:12 timoguin there's a minion config option for that: startup_states: highstate
16:13 timoguin although that's not strictly for when a minion joins a master, but when the minion is started
16:14 salt_whitebelt when running 'salt window.vm cmd.run powershell.exe get-process' on windows 7 minion, isn't it suppose to show you some result on the centos master terminal?
16:15 timoguin salt_whitebelt, generally, yes. but there's a default timeout for the CLI, a few seconds i believe
16:15 timoguin you can append --timeout=300 (or any other value)
16:15 timoguin that's sometimes necessary. windows minions tend to respond a bit slower
16:15 analogbyte jcockhren: where do I place a custom runner? saltutil.sync_all does not take care of that
16:16 Ahlee analogbyte: runners don't sync to minions
16:16 timoguin salt_whitebelt, if that doesn't help there may be something else going on. you can try adding a '-l debug' to the salt command to see if that helps
16:17 Ahlee analogbyte: http://docs.saltstack.com/ref/runners/index.html
16:17 jalbretsen joined #salt
16:17 Ahlee analogbyte: define a runner_roots: in your master config
16:17 Ahlee drop your custom runners there
16:18 mgw joined #salt
16:19 analogbyte Ahlee: yeah, of course they don't, but this was the place I started looking for some runner discovery stuff... thanks for pointing out the setting ;)
16:19 salt_whitebelt so i add 'salt window.vm cmd.run powershell.exe get-process --timeout=300' ?
16:20 Ahlee no problem
16:21 analogbyte Ahlee: the setting is called runner_dirs actually, but I found it ;)
16:21 Ahlee hrm. wonder if that's changed as I have it commented out in mine as runner_roots
16:21 Ahlee or another case of why yaml sucks
16:22 ekristen sroegner: that complex search doesn’t seem to work for me
16:23 sroegner ekristen: have an example?
16:23 ekristen sroegner: well I think the mine.get isn’t matching my environment grain for some reason
16:24 ekristen it might be the way I’m calling it from the cli, but it doesn’t work in my jinja template either
16:24 ekristen sroegner: sudo salt '*' mine.get 'G@environment:dev' network.interfaces grain
16:25 sroegner that's not supposed to work, no
16:25 sroegner if you use the G@ prefix you need to qualify this as a compound match
16:26 sroegner do you set the environment grain
16:26 sroegner ?
16:28 jakubek how to print in jinja for loop everything in same line with spaces? http://pastebin.com/0ep2E4Dj
16:28 jakubek now its prining but without spaces between words
16:30 dave_den jakubek: use {%-
16:30 dave_den derp,
16:31 dave_den jakubek: use {{ pillar['rules']|join(' ') }}
16:31 jakubek dave_den: worked, thanks ;-)
16:31 jakubek just {%-, worked
16:31 ekristen sroegner: sorry, phone
16:31 jakubek beczause i use loop in one line
16:31 ekristen sroegner: I did define the environment grain
16:31 ekristen sroegner: how do I specify it as a compound match?
16:32 dave_den s/grain/compound/
16:32 sroegner the last argument in your example currently says 'grain'
16:32 jakubek dave_den: thanks :-)
16:32 ekristen oh, duh
16:32 ekristen ;)
16:32 sroegner replace that with compound
16:32 dave_den jakubek: no prob
16:33 psyl0n joined #salt
16:33 ekristen for some reason my brain was thinking that that last argument “grain” was for what to return
16:33 ekristen thanks!
16:33 BogdanR How should I handle the fact that on Amazon instances have a hostname generated from the IP address?
16:33 ekristen what is the difference between %} and -%} and {% and {%-
16:33 timoguin salt_whitebelt, actually, there is a shell option for cmd.run. try this: salt window.vm cmd.run get-process shell=powershell --timeout=300
16:34 BogdanR I would like to have a readable hostname which would make it easy for me to know it which machine is salt issuing commands.
16:34 dave_den ekristen: {% causes whitespace, {%- does not
16:34 BogdanR On EC2 the hostname is configured by Amazon's DHCP server
16:34 ekristen dave_den: sweet, do I have to use -%} if I use {%- ?
16:34 dave_den http://jinja.pocoo.org/docs/
16:34 Sypher BogdanR you can declare an id for the minion to use
16:35 Sypher by default it uses the fqdn, but you can set it to anything you'd like as long as you have separate id's per minion
16:35 Sypher its a setting in the minion config
16:36 dave_den ekristen: no. as long as you use '-' at either the start or end of a block it will strip whitespace
16:36 ekristen cool
16:36 dave_den i prefer to put '-' in both places though
16:36 ekristen thanks!
16:37 BogdanR Sypher: But I would like to have things as automated as possible
16:38 BogdanR So I would have an AMI with salt installed standard and they with the hostname the key would appear in salt-key
16:39 sroegner BogdanR: I really think your requirements match salt-cloud
16:39 timoguin BogdanR, I use salt-cloud to launch new instances, and I define the minion id when I launch them.
16:40 timoguin it will create the instance via the providers API, bootstrap salt, generate and copy over the minion key, and fire off any states i define
16:41 BogdanR I see. So if I define a minion ID the hostname will not be used?
16:41 timoguin correct
16:42 luimemee the minion initialize the connection to the master. is it possible to to it in another way so the master can push or init the conenction ?
16:42 timoguin luimemee, salt-ssh can do that, but otherwise, I don't think so.
16:43 luimemee timoguin, is there a way to use salt-ssh from python i mean other than command line ?
16:43 BogdanR Thanks a lot. I will find a sollution based on the minion_id
16:43 timoguin luimemee, I'm not sure, sorry
16:44 ekristen thanks dave_den
16:44 luimemee k thx anyway
16:45 dave_den welcome
16:47 gnu-man joined #salt
16:48 patrek joined #salt
16:49 KyleG joined #salt
16:49 KyleG joined #salt
16:50 gmtech Hey.. do ppl tend to have formulas as separate git repos, or do lump them altogether in one. And can I ask why?  (noticing the conversation from earlier) @timoguin
16:51 jdenning joined #salt
16:52 timoguin gmtech, I'm not sure how most people are handling it. I just use them as a reference and copy the contents into a single git repo that holds all my states.
16:52 timoguin They're designed so people can just stick each formula they want to use into their gitfs_remotes
16:52 Ahlee gmtech: you referring to salt-forumlas?
16:52 gmtech Yeah
16:53 Ahlee i copied one locally, as I would never let a salt master near a repo I didn't control
16:53 gmtech Sure, but do you still have it as a separate git repo, even though you copied it locally - so hosted on your own git server or something like that?
16:54 Ahlee Correct.
16:54 Ahlee It's hosted on our internal git server, since none of my servers have access to the internet
16:54 Ahlee even if they did, again, I would not trust somebody else with the ability to change something on my servers
16:54 gmtech What are the advantages of having 1 gitrepo per formula, rather than 1 gitrepo to version the whole lot?
16:55 Ahlee philisofically or realistically?
16:55 cdsrv joined #salt
16:55 Ahlee it's just different ways of doing things.  Some people like one large repo, some people liek small repos
16:55 gmtech realistically I guess. Example: If I have a change that affects 4 formulas, thats 4 places to commit (and/or possibly merge) rather than 1
16:55 troyready joined #salt
16:56 gmtech OK so its preference really
16:56 jacksontj joined #salt
16:56 gmtech Just trying to understand, quite new to the salt ways
16:59 bhosmer joined #salt
16:59 jslatts joined #salt
17:00 esoel_ joined #salt
17:02 kermit joined #salt
17:08 iri- joined #salt
17:10 ekristen I have multiple environments in my salt config, but I am not seeing my dev environment pillar data getting added to a minion
17:12 NotreDev joined #salt
17:14 carmony ekristen: minions will cache their pillar data, did you do a pillar_refresh?
17:15 ekristen carmony: no, how do I do that, also is there a way to test to see if my minion is picking up the right sections in my pillar top.sls?
17:16 carmony ekristen: http://docs.saltstack.com/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.refresh_pillar
17:16 q1x joined #salt
17:16 Flusher joined #salt
17:17 Flusher joined #salt
17:17 gnu-man joined #salt
17:18 ekristen carmony: yeah, it still isn’t picking up my dev env pillar info
17:19 carmony k, let me think
17:19 carmony you should be able to have the minion tell you what envs it has
17:19 ekristen carmony: I think it might have been a typo in my dev pillar file, I’m checking
17:19 ekristen carmony: that’d be great, how do I do that?
17:21 schmichael left #salt
17:21 tessellare joined #salt
17:21 dave_den there's no easy way to see how a minion matches pillar environments. for checking what envs it will match in the states top.sls, you can run 'salt-call state.show_states' from the minion
17:23 carmony dave_den: would what envs be listed in the grains?
17:23 joehillen joined #salt
17:24 jgiraldo_ So when I service salt-master stop, ksoftirqd shoots up to 100% utilization
17:24 bhosmer joined #salt
17:24 q1x anyone here experienced with building docker containers using salt?
17:24 dave_den carmony: not unless they were explicitly set via grains
17:24 davet joined #salt
17:24 dave_den there are multiple ways to define what env you want to use for things. if you set env via grains it's similar to setting the env in the minion config
17:27 ajw0100 joined #salt
17:28 davet joined #salt
17:28 diegows joined #salt
17:28 bhosmer_ joined #salt
17:28 carmony then that might just be a useful tool to set some grains in a env for the purpose of identifying them
17:28 fllr joined #salt
17:28 carmony I haven't used envs much
17:29 q1x I was wondering if it would be possible to automatically register (salt-key) salt minions that live in the docker containers if the container was created using salt
17:29 fllr Hey guys. Which one is the latest documentation on salt cloud? this: http://docs.saltstack.com/topics/cloud/index.html or this: http://salt-cloud.readthedocs.org/en/latest/
17:29 q1x and what the best way of doing that would be
17:29 JordanRinke Morning
17:30 timoguin fllr, pretty sure they are both generated from the same source.
17:30 fllr Ah, cool.
17:31 ekristen q1x: are you wanting salt to “build” images or launch containers
17:31 timoguin the docs/ folder in the salt source
17:31 martoss1 joined #salt
17:32 q1x ekristen: I figured out both of those already :-)
17:32 q1x ekristen: Now I'm lokking to put minions in the containers and have those contact the master
17:32 q1x *looking
17:33 ekristen q1x: use the dockerfile
17:33 q1x ekristen: but I don't want to manually add a minion to the master every time
17:33 ekristen have a run command that bootstraps salt-minion and puts a /etc/salt/minion file into place
17:34 higgs001_ joined #salt
17:34 ekristen q1x: you’ll have to turn on open_mode, or write a script that launches the minion, then goes and looks at the docker containre filesystem to find what key it used and call salt-key -a <keyid>
17:34 ekristen err launches the container
17:34 q1x open_mode is a no go, it's all public internet stuff
17:34 gmtech I'm surprised that salt['pillar.get']('foo', 'bar') is the recommended safe way to get data out of pillar (with a default). This seems quite verbose to me. Why not assign salt['pillar.get'] to a variable e.g. lookup, and then call lookup('foo', 'bar'). This seems slightly better ?
17:35 q1x so...guess I need to script it then
17:35 q1x how does that mechanism work in Salt-cloud?
17:35 ekristen as far as I know yes
17:35 ekristen same way
17:35 ekristen salt-cloud run from the master
17:35 ekristen although it generates the minion-key locally then ssh it over to the vm and accepts it in the master
17:35 ekristen I wouldn’t advise that due to how docker containers work
17:36 gadams9991 joined #salt
17:36 q1x ekristen: Ah, I was looking for something like that!
17:36 q1x but tha doesn't exist then :-/
17:36 q1x *that
17:37 danielbachhuber joined #salt
17:37 q1x ekristen: why would you advice against it?
17:37 q1x the ssh way I mean
17:38 q1x Perhaps I could use salt-ssh for it?
17:38 * q1x is still quite new to salt
17:38 ekristen you’d have to install openssh in your container too, and use supervisord to run multiple apps inside the container, ie ssh, your app, etc
17:38 adizam joined #salt
17:38 ekristen q1x: what are you trying to do with docker
17:38 ekristen I’m new to salt, but not docker
17:39 q1x ekristen: I'd like my apps to be isolated from each other, but I don't like the VM overhead. Docker seemed to be a nice way to work around that, plus it's fast!
17:40 ekristen great, that is its purpose ;)
17:40 q1x but ideally, I would like them to be 'immutable'
17:40 q1x so I was thinking of having a general purpose image that just intalls a salt-minion
17:40 ekristen so using ssh with docker containers is doable but you have to use supervisord or have an entrypoint that looks like /bin/openssh-server &; /run/my/app
17:41 q1x and then have salt-states handle the rest
17:41 q1x ie: install mysql, install apache, etc
17:41 q1x use docker mounts for data directories
17:41 gnu-man joined #salt
17:42 q1x then I can just kill a container, create a new one and have a fully updated instance that is freshly build and using the data directories
17:42 q1x at leat, that was my plan :-)
17:42 q1x *least
17:43 q1x ekristen: does my plan sound doable/sane?
17:43 n8n joined #salt
17:43 kickerdog joined #salt
17:44 adizam Question
17:45 adizam Trying to jumpstart a bit of documentation reading here
17:45 adizam So I've created a standalone minion.. vagrant (vmware).. anyway. Through various attempts/etc.. I've got it working as I want.. but I'm a little confused
17:45 ekristen q1x: sure you can do that, not what I would do personally
17:45 adizam I just want to know *what* this one part of my .sls file is
17:46 adizam For instance.. I have postgres installing.. pretty simple. postgresql: pkg: - installed
17:46 adizam No configuration/etc.. but.. how does salt know what postgresql is. Does it install it via apt-get?
17:46 ekristen q1x: one of the best features of docker is the Dockerfile and building out pre-built images then all you have to do is call docker run and you have a full functioning instance in a few miliseconds
17:46 timoguin adizam, yes, it installs the package via apt
17:46 elfixit joined #salt
17:46 timoguin it will chose the package management system on the target OS
17:46 adizam timoguin: Thats what I thought. Was just making sure. Again.. learning :p
17:47 q1x ekristen: yeah... but how do you manage stuff inside the container?
17:47 adizam Now to figure out how to get ruby2 installing. Mm. Off to it
17:47 ekristen q1x: lets go PM for this discussion
17:47 q1x ekristen: sure
17:50 JesseCW joined #salt
17:50 JesseCW left #salt
17:51 toastedpenguin anyone here using salt for AWS VPC?
17:51 toastedpenguin looking for advice on how to best use it in an enviroment that is exclusively in AWS
17:51 timoguin toastedpenguin, I know many people are. I've just recently started getting my AWS stuff working, but I've not setup VPCs yet
17:52 toastedpenguin last place I was using salt we had dedicated data center
17:52 timoguin do you have any specific questions?
17:53 timoguin i'm using salt-cloud to manage my instances
17:53 toastedpenguin where is the master, inside the cloud env?
17:53 toastedpenguin as an instance?
17:53 timoguin yes
17:54 timoguin my master is actually on DigitalOcean, but same concept. i'd put it inside a VPC and have it listen on the internal subnet
17:54 adizam Am I doing this right: https://gist.github.com/adizam/ad4467323eae0ba46a6f
17:54 adizam J/w
17:55 jgiraldo_ would configuring a batch of approx 1000 minion to run highstate over a period of 10 minutes overload a saltmaster running 12 cores and 16GB?
17:55 adizam Everything works except ruby so far.. working on that :p
17:55 JesseCW joined #salt
17:56 kickerdog adizam: are you running on redhat or debian? Usually with ruby I have to specify the full package name from yum or apt-get.
17:57 adizam ubuntu, vagrant run
17:57 pb__ joined #salt
17:57 timoguin adizam, yea you need the full package name that apt uses
17:57 kickerdog try using ruby1.9.1
17:58 adizam I need ruby2
17:58 adizam Do you specify..
17:58 jacksontj joined #salt
17:58 adizam ruby: pkg: version: 2.0.0
17:58 kickerdog yeah
17:58 adizam or ruby2.0.0: pkg: etc
17:58 Ahlee ok, I have a python script that now parses /proc/net/bonding/bond0.  if what I see in http://docs.saltstack.com/topics/development/external_pillars.html, I effectively just wrap this all up as a def ext_pillar(boilerplate):, then return my_pillar{}.  I can then access this data by -I <name of file ext_pillar> is defined in?
17:58 JasonSwindle joined #salt
17:58 timoguin adizam, whatever the package name is in that PPA is what you need to list
17:58 kickerdog do a "apt-cache search ruby" and grab to version you want
17:58 Ahlee once sync'd, that is
17:58 adizam The package name is ruby2.0
17:58 adizam Is that legal in the sls file?
17:58 kickerdog yeah
17:59 gadams999 joined #salt
17:59 adizam Oh you're right -_-
17:59 adizam Last question I promise.. where is a good resource (I know, the docs, but where in the docs) to read about executing custom commands
18:00 adizam (I need to install modules/gems via npm/bundle post-provision)
18:00 JasonSwindle JordanRinke:  Morning!
18:00 kickerdog adizam: like this -> http://stackoverflow.com/questions/16362423/how-do-i-make-one-custom-state-dependent-on-another
18:00 dave_den adizam: http://docs.saltstack.com/ref/states/all/salt.states.cmd.html
18:01 higgs001 joined #salt
18:01 adizam dave_den: perfect
18:01 adizam kickerdog: That is interesting too
18:02 dancat joined #salt
18:02 timoguin adizam, you may also want to check out the gem and npm modules and states
18:02 timoguin rather than having to use cmd.run
18:03 njpatel joined #salt
18:03 adizam timoguin: What do you mean? (I'm still learning)
18:04 timoguin adizam, I'm guessing you want to make sure certain gems and npm packages are installed?
18:04 ekristen how can I see what state file is calling a command?
18:04 timoguin for gems, there's the gem state: http://docs.saltstack.com/ref/states/all/salt.states.gem.html
18:05 ekristen each time highstate runs I see apt-get -y update
18:05 ekristen running
18:05 timoguin and for npm, the npm state: http://docs.saltstack.com/ref/states/all/salt.states.npm.html
18:05 adizam timoguin: Yes. I have ~9 npm packages that need to be installed globally. The remainder (and gems) can be installed via cmd.run (bundle install, npm install).. as I don't want my devs having to update a provisioning file everytime a new dependency is added to the project. Just the larger ones
18:05 adizam *reading*
18:07 gnu-man joined #salt
18:09 ekristen timoguin: if I have a pkg.installed in a state does that call apt-get -y update to be run each time?
18:09 timoguin ekristen, I'm not sure. I think that may run on each highstate regardless.
18:09 timoguin at least i've noticed that in mine
18:11 xmltok joined #salt
18:14 dimeshake therer's pkg.installed and pkg.latest i think
18:14 dimeshake the latter would ensure an update, i believe
18:15 Ryan_Lane2 joined #salt
18:16 ekristen can’t I do this set dev = pillar.get(‘dev’, {}) and then dev.get(‘app:port’)
18:16 dave_den apt-get update runs once per highstate, even with multiple package.installed states.
18:16 ekristen where the state file looks like dev: app: port: 8000
18:16 dave_den apt-get update will run multiple times only if there is a package.installed state that specifies '- refresh: True'
18:17 gasbakid joined #salt
18:18 dave_den ekristen: no, you can't do that
18:18 dave_den but you can do
18:19 dave_den set dev = pillar.get(‘dev’, {}) and then dev['app']['port']
18:19 dave_den but if the pillar.get returns the empty hash it will blow up
18:19 jgiraldo_ Is there any reason why a handful of salt-master threads will be pegging a whole core each at 100% cpu usage?  I have a group of 1000 minions checking into the master
18:19 kermit joined #salt
18:19 dave_den so you can do a condition check after the pillar.get
18:20 Ahlee jgiraldo_: any of those 1000 minions doing any file sync?
18:20 Ahlee jgiraldo_: and congrats, I can't get above 120 minions before the master falls over
18:21 jgiraldo_ Thanks!  Tho apparently I haven't gotten it to work because salt is acting up
18:21 Ahlee jgiraldo_: look at https://github.com/saltstack/salt/pull/9374?source=cc if you haven't applied that already, that helped a little for me
18:21 jgiraldo_ NOt sure what you mean by file sync
18:22 ndrei joined #salt
18:22 Ahlee jgiraldo_: if the minions are asking hte master for a file
18:22 jgiraldo_ Just a few config files
18:22 Ahlee out of interest, how beefy is your master?
18:22 jgiraldo_ a few hundred kb tops
18:22 Ahlee and are you using gitfs?
18:23 jgiraldo_ 12 core, 12 GB RAM
18:23 jgiraldo_ its in a VM tho
18:23 jgiraldo_ and the configs are stored locally on the master
18:23 Ahlee 8 core/6 gigs here, i'm not seeing any memory pressure so i haven't increased
18:23 Ahlee gotcha. that's my next step
18:24 jgiraldo_ my next step would be to host the config files on gitfs?
18:24 Ahlee if i hit my 100ish minions at once (8 or 16 worker threads), my salt master becomes completely unresponsive until the run finishes
18:25 nineteeneightd joined #salt
18:25 Ahlee i wouldn't until they roll out the pygit2 implemntation
18:25 Ahlee have you tried stracing one of the threads while it's pegging the CPU?
18:25 jgiraldo_ Yeah I found that running highstate on the master runs, but almost half the minions arent even contacted
18:26 jgiraldo_ so instead i set the minions to check in at a 10 minute interval
18:26 Ahlee that's what I see, along with ZMQ timeouts
18:26 jgiraldo_ Then what's best practice?  have one syndic per 100 minions?
18:26 Ahlee no idea.
18:27 Ahlee I'm currently working with their engineers
18:27 jgiraldo_ shoot
18:27 Ahlee but, for the most part i'm in a holding pattern as management doesn't want to abandon gitfs
18:27 Ahlee but if you're seeing it with local files, i got nothing
18:27 jgiraldo_ ok, well thanks for the help
18:27 Ahlee i mounted /var/cache/salt as tmpfs to keep it in memory
18:28 Ahlee jgiraldo_: strace the proc though and verify if it's just stat()ing files
18:28 jacksontj joined #salt
18:28 Ahlee if so, try the github pull request i linked
18:28 Ahlee that did cut my load down significantly
18:28 jgiraldo_ ok will do
18:29 jgiraldo_ thanks again ahlee
18:29 Ahlee and it's from a team that somehow runs thousands of minions successfully
18:30 rlarkin joined #salt
18:31 jgiraldo_ At this point I'll settle for running a few score hundred
18:31 danielbachhuber- joined #salt
18:32 jgiraldo_ not score
18:32 jgiraldo_ just hundred
18:32 jgiraldo_ lol
18:32 gadams9991 joined #salt
18:32 Ahlee lol, slight difference there
18:34 kermit joined #salt
18:35 jimallman joined #salt
18:35 gnu-man joined #salt
18:36 srage joined #salt
18:41 ekristen dave_den: can I do pillar.get(‘dev:app:port’, 8000)
18:43 Whissi joined #salt
18:47 mpanetta_ joined #salt
18:49 thrashr888 joined #salt
18:49 jdenning basepi: I have a couple of extensions that I'm about to release (an external pillar, a returner and a tops module) - to contribute upstream, should I fork the entire repo and do a pull request from that?  Also, should I make separate requests for each module?  (They are all AWS oriented)
18:50 dave_den ekristen: no, you cannot do that. pillar is a normal python dict
18:51 dave_den ekristen: you would have to use the pillar module from the salt dict
18:51 bhosmer joined #salt
18:51 dave_den salt['pillar.get']('dev:app:port', 8000)
18:51 bhosmer joined #salt
18:52 ekristen dave_den: is there a problem doing that so I can have a default value?
18:52 dave_den use the salt['pillar.get'] method ^^
18:52 dave_den that will do what you want
18:53 Marion joined #salt
18:53 ekristen no I mean is it bad to use salt[‘pillar.get’] vs pillar.get
18:53 ekristen obviously pillar.get won’t get me what I want
18:53 cachedout jdenning: Here's the normal process. https://github.com/saltstack/salt/blob/develop/HACKING.rst
18:53 ekristen I guess I’m asking is there a trade-off
18:54 jdenning cachedout: Thanks
18:54 cachedout jdenning: Sure thing. Let me know if you have questions and thanks for the contributions! :]
18:54 Mua joined #salt
18:54 Nexpro1 joined #salt
18:54 Ahlee anybody actaully written an ext_pillar?
18:55 ConceitedCode joined #salt
18:55 jdenning Ahlee: yup, I did earlier this week - that's one of the things I was mentioning about releasing this week :)
18:55 morte joined #salt
18:55 apergos uh so a newby-ish question about salt + -b (batch) option... it seems it doesn't wait for the response to come back past the default timeout but just prints a job id if the command hasn't completed?
18:55 jdenning (uses DynamoDB as a data source)
18:55 n8n joined #salt
18:55 forresta joined #salt
18:56 Ahlee jdenning: awesome. So I dropped my python that parses a file on disk and returns a dict, i tossed it in pillar_roots, but when I try to saltutil.refresh_pillar i get None back
18:56 dave_den ekristen: the trade-off is that pillar.get is direct python code inside the jinja engine. salt['pillar.get]' has to execute the salt pillar module and run a dictionary traversal over the pillar dict. it is only a very slight tradeoff in speed.
18:56 jdenning ekristen: Nope, that's the intended use of salt['pillar.get']
18:57 jdenning (indeed, there is a minimal performance penalty)
18:57 srage joined #salt
18:57 ekristen jdenning: fair enough
18:57 ekristen thanks dave_den
18:57 dave_den yep
18:58 gadams9991 left #salt
18:58 jdenning Ahlee: you need to set the path for the extension in the master config..one sec and I'll find the attrib name
18:59 apergos ah seems to be a known buglet
18:59 apergos https://github.com/saltstack/salt/issues/8597
18:59 jdenning Ahlee: in your /etc/salt/master :
18:59 * apergos wanders off again
18:59 Ryan_Lane2 joined #salt
19:00 jdenning extension_modules: /srv/salt/ext (or wherever you have your extensions)
19:00 jdenning then in /srv/salt/ext create a subdir 'pillar', and put your code there
19:00 jdenning so code should live in  (e.g.) /srv/salt/ext/pillar
19:00 sporked_ joined #salt
19:02 amckinley joined #salt
19:04 Ahlee k. restarting master
19:04 Ahlee then just refresh_pillar/
19:05 robbyt joined #salt
19:06 Ryan_Lane2 joined #salt
19:08 jdenning Ahlee: make sure you have the logs set to debug, and are watching when you try to run the code (e.g. w/ salt '*' pillar.items)
19:09 mpanetta joined #salt
19:09 jdenning (if you aren't watching the logs, you won't see it if your custom code raises exceptions)
19:09 Ahlee neat, it at least compiled down a .pyc
19:09 matanya joined #salt
19:10 Ahlee so, progress
19:13 jdenning cool :)
19:13 scott_w joined #salt
19:16 bhosmer_ joined #salt
19:17 ndrei joined #salt
19:17 beach joined #salt
19:18 tyler-baker joined #salt
19:19 adizam hm
19:26 njpatel joined #salt
19:28 NotreDev joined #salt
19:28 dmwuw joined #salt
19:29 kermit joined #salt
19:29 sroegner joined #salt
19:30 foxx joined #salt
19:30 tyler-baker joined #salt
19:30 tyler-baker joined #salt
19:32 zooz joined #salt
19:33 zooz joined #salt
19:33 druonysus joined #salt
19:33 abhi09 joined #salt
19:35 adizam Using the approach I've taken (new to salt).. what would be the best direction for installation of global npm packages? https://gist.github.com/adizam/c5d317ec9cee655baece
19:36 ajw0100 joined #salt
19:37 timoguin adizam, the npm state would be best. something like, coffeescript:  npm.installed
19:37 timoguin for each package you want to install
19:37 adizam Hm
19:37 adizam I just tried that with...
19:37 forresta adizam, ok you're thinking about this in a weird way, my first question is, why does all of this live in a single state?
19:38 adizam forresta: Because I'm only now learning what 'state' is in saltstack
19:38 mohan__ joined #salt
19:38 forresta adizam, ok cool
19:38 bhosmer joined #salt
19:38 forresta let's look at an example :D
19:38 adizam forresta: I originally had a few different sls files, for specific needs (node, ruby, project, etc)
19:38 forresta yea that's the way to go!
19:38 forresta https://gist.github.com/adizam/c5d317ec9cee655baece
19:38 forresta https://github.com/terminalmage/djangocon2013-sls
19:38 adizam Right now I'm just trying to get it to 'work'.. I can organize later
19:38 forresta sorry wrong ling
19:38 adizam Consider this my spike :p
19:38 forresta ok, so what you could do, is if you have those different states, just use 'include'\
19:38 adizam keep in mind, this is only to provision a vagrant box for my developers
19:39 adizam We've been using puppet for a while.. I hate puppet
19:39 forresta this will allow you to put multiple states, into a single like you have now, without that duplication
19:39 adizam Okay so.. in my top.sls file
19:39 forresta adizam, I understand that is it only for vagrant, but the more modular you keep things, the easier it will be to expand on later.
19:39 adizam base: '*': - onestate - anotherstate -anotherstate
19:40 forresta that's a good way to do it
19:41 mohan__ Hello, I am a newbie getting started on Salt. I am running into some problems with puppet.run. Can anyone please help
19:42 adizam I still have no idea what a pillar is :p
19:42 diegows joined #salt
19:42 timoguin adizam, pillar is just key-value pairs you can define that you can use in SLS files
19:42 timoguin I have my apache configs defined in pillar. And then the SLS files are templated
19:42 adizam Ah. Gotcha
19:42 timoguin for example
19:42 forresta clearly, the best explanation is private global variables, duh
19:43 adizam Yeah I get it
19:43 mohan__ "Under the hood, pillar is just a python dict, so python dict methods such as get and items can be used."
19:43 adizam Alrighty. So organization standards aside..
19:43 adizam The best approach to installation of npm packages.. packagename: npm.installed
19:44 mmlb joined #salt
19:44 forresta yea, just make sure that npm is installed before you do that
19:44 adizam The lack of a dir directive implies it will be installed globally, yes?
19:44 forresta so install npm, then require the npm package in your npm.installed
19:44 adizam require the npm package?
19:44 forresta well, there is a dir option
19:44 forresta yea
19:45 forresta so how are you gonna use npm, without npm installed right?
19:45 timoguin yea, leaving out that option means global
19:45 adizam packagename: npm.installed doesnt 'require' npm ?
19:46 forresta no
19:46 adizam I need to explicitly require it, correct?
19:46 forresta yes
19:46 adizam Gotcha
19:46 forresta you need to make sure it's installed
19:46 adizam Is there any kind of reverse syntax.. rather than
19:46 forresta so npm: \n pkg.installed
19:46 adizam module: npm.required
19:46 oeuftete So what is the deal with salt-cloud?  Getting really confused... in the docs, but doesn't appear to be in the repo anymore?
19:46 forresta then you do your npm.installed
19:46 adizam I coudl do something like npm_modules: moduleone, moduletwo, modulethree
19:46 adizam know what I mean?
19:46 timoguin oeuftete, it's been merged into salt
19:46 forresta oeuftete, if you're using the release candidate, it's now part of salt, if you are using an older release of salt, please install from pip
19:47 forresta adizam, I don't know if you can pass multiple options with npm.installed like that.
19:47 iri- joined #salt
19:47 forresta the example over here shows each one having it's own ID http://docs.saltstack.com/ref/states/all/salt.states.npm.html
19:48 ajw0100 joined #salt
19:48 oeuftete hmmm... I have 0.17.4 via the ubuntu ppa.  I've just been looking at the saltstack/salt repo too and I don't see what I'm looking for in there
19:49 forresta yea 0.17.4 uses 0.8.11: http://docs.saltstack.com/topics/cloud/troubleshooting.html#generic-troubleshooting-steps
19:49 forresta you can get 0.8.11 here: https://pypi.python.org/pypi/salt-cloud
19:49 oeuftete forresta: thanks.  I'll try one way or another.
19:50 forresta yea np
19:50 salt_whitebelt how do i call a powershell script not powershell executable with commands syntax to run from master (centos) server using "salt windows.minion cmd.run powershell?"
19:52 timoguin salt_whitebelt, you should be able to do this: 'salt windows.minion cmd.run Script.ps1 shell=powershell'
19:53 timoguin there's also cmd.script
19:53 timoguin but cmd.run should do the trick
19:56 MTecknology I have a lot of boxes that take a LOOOOONG time to start running state.highstate. Once they start, they truck right along at an expected speed. They hang right after this -- http://dpaste.com/1551368/
19:57 sroegner joined #salt
19:57 CaptTofu joined #salt
19:57 CaptTofu hi salt folks!
19:58 kermit joined #salt
19:58 adizam forresta: https://www.dropbox.com/s/4ztijs1jwstxxu6/Screenshot%202014-01-14%2013.58.32.png
19:58 adizam better? :p
19:59 giantlock_ joined #salt
19:59 forresta adizam, Yea that's way better!
19:59 forresta then you won't want to kill yourself in 6 months like when I had to rewrite a ton of puppet
19:59 dave_den is registry.npmjs.org borked or what
19:59 srage joined #salt
19:59 adizam Luckily this is only for vagrant provisioning.. I'd be a lot more anal if this was used in any type of deployment
20:00 forresta heh
20:00 adizam Which.. with how much engine yard is pissing me off lately
20:00 adizam Might end up being the case :p
20:01 felixhummel joined #salt
20:08 JasonSwindle forresta:  BTW, I am now staying mid-day friday as well at SaltConf
20:08 JasonSwindle Woot
20:08 forresta nice
20:09 forresta basically you're flying out friday, instead of mid day thursday then?
20:09 gnu-man joined #salt
20:09 JasonSwindle Yep
20:09 JasonSwindle An issue came up, and I needed to bump my fly time out later.
20:10 forresta nice
20:10 forresta I was torn between going to work late friday, or using another day off, ended up just using the day off.
20:12 MTecknology There's no log level above debug, is theer?
20:13 MTecknology below?
20:14 JasonSwindle Trace?
20:15 MTecknology ah, true
20:15 JasonSwindle I never found it that helpful
20:15 MTecknology trace doesn't give me any more indication why it's hanging here... :(
20:15 JasonSwindle debug always got me by...
20:15 JasonSwindle yep, ditto.
20:15 salt_whitebelt timogui: don't i need to specifty the path where the script reside on the master (centos)
20:16 Mua joined #salt
20:17 njpatel joined #salt
20:17 dancat joined #salt
20:20 MTecknology strace output is showing a LOT of these when it's hanging...    read(4, "de\0\0\0\0\4\0\0\0003710\1\0\0\0*\335\1\0\0\v\0\0\0\10\0\0\0l"..., 16284) = 128   poll([{fd=4, events=POLLIN}], 1, 60000) = 1 ([{fd=4, revents=POLLIN}])    It hangs right at that '=' and then eventually displays the rest of that line
20:20 timoguin salt_whitebelt, yea, that's what you'd put where i wrote "Script.ps1"
20:20 timoguin just put the path there
20:20 wacko joined #salt
20:21 MTecknology I just realized that this is doing a massive poll of users
20:22 MTecknology there's over 30,000 user accounts... Why would it be trying to do that?
20:22 wacko hi all, i am trying to debug an issue with a pillar dictionary and am having a hard time identifying the exact issue.  sometimes errors have all the info that i need to troubleshoot, but other times, not so much.  i am looking for advice on how to debug this given the following output:  http://pastebin.com/WxTMqQxz
20:23 wacko is "line 46" likely the line that is having a problem in my salt state?
20:23 darless_ joined #salt
20:24 srage joined #salt
20:25 wacko i am reading this as an issue with the pillar dict not being complete, and so the salt state that needs the missing data is complaining.  i'm just trying to get more info on where this is actually happening, and wondering if folks are using different approaches when troubleshooting issues like this
20:26 ekristen why would {{ variable }} not get replaced in a jinja template?
20:26 timoguin ekristen, what ends up in the file?
20:26 Mua joined #salt
20:26 ekristen {{ variable }}
20:26 timoguin is it a file.managed state?
20:27 timoguin you have to specify - template: jinja
20:27 ekristen yeah I did that
20:27 ekristen other variables get replaced
20:27 ekristen but some don’t
20:27 cachedout MTecknology: Are you using eauth by chance?
20:27 timoguin hmm. dunno without seeing the states.
20:28 MTecknology cachedout: If I am, I'm not aware of it
20:28 cachedout MTecknology: K
20:29 cachedout Are you using states to manage users on the minions that are slow?
20:29 MTecknology only some states look at users/groups; it shouldn't need to cache all users/groups, though...
20:30 cachedout MTecknology: Let me code-dive for a bit.
20:30 MTecknology cachedout: will do- shall I create an issue?
20:30 MTecknology I was in the process of doing that before you replied
20:30 cachedout MTecknology: Yes. Please include state files being applied.
20:31 MTecknology I'll try to use my best judgement with those... there's >100 states being applied
20:32 Mua_ joined #salt
20:33 cachedout MTecknology: OK. That's fine then. We don't need them all. I'm mostly interested in anything that might be iterating users, so use your judgement there.
20:34 MTecknology no states that I have actually iterate through users, they're all explicitely defined
20:34 MTecknology explicitly*
20:35 cachedout MTecknology: Yes, but the way the user state works (I think) is that it still needs to search the password file each time it checks for the state of a user. We might need to optimize that.
20:36 cachedout So, if you're checking a thousand users, it's doing a thousand searches through the file instead of loading it once into memory, possibly. I need to keep digging.
20:37 psyl0n joined #salt
20:38 basepi jdenning: if it was not already answered earlier, I would recommend forking the repo and contributing in pull requests.  As far as a single pull request vs many, it depends on how related the features are.  For example, if someone were to write a mysql module and state, I would say same pull req.  But any less related, I would recommend separate.
20:38 jdenning basepi: cool, thanks!
20:38 basepi jdenning: no problem, sorry for my slow reply.
20:39 WarheadsSE left #salt
20:39 thrashr888 joined #salt
20:39 MTecknology cachedout: they're not local users... all the users it's searching are AD accounts (nslcd)
20:40 cachedout Yup, that makes perfect sense actually. One moment.
20:40 thrashr888 joined #salt
20:42 MTecknology I'm grabbing a state
20:44 MTecknology cachedout: https://github.com/saltstack/salt/issues/9743 <-- I was able to do all of that and I'm still watching the strace do user look-ups
20:45 kainswor joined #salt
20:45 alunduil joined #salt
20:46 MTecknology I also noticed that the entropy available on many servers is incredibly low so the keys take a LOOOONG time to generate. It's a one time issue, but proves to be a headache. Are there any quick/easy solutions like telling it to use /dev/urandom instead of /dev/random?
20:46 n8n joined #salt
20:47 MTecknology cachedout: If I stop nslcd, BOOOOM, it goes ZOOMY
20:48 cachedout MTecknology: Ah, interesting.
20:48 oeuftete joined #salt
20:48 fxhp is it possible to set a default pillar in '*' and then overide it for various sub domains ?
20:48 MTecknology I'm wondering how it'll handle the stuff that requires that....
20:52 djinni` joined #salt
20:53 cachedout MTecknology: I have to run to a meeting right now but basically the issue here is that lookups are happening for each of those users you're managing and those lookups are blocking. In cases where we have local users, we just pull down the user list from the passwd file once and iterate over it in memory. I'm not sure if that's possible in your setup so that's something we'd need to investigate. At any rate, I hope that helps explain what's likely
20:53 cachedout happening here.
20:53 analogbyte the documentation is not entirely clear on this, but it doesn't work for me: I can't put multiple states (top level) in a reactor sls file? I have to use multiple sls files like in the first code example here: http://docs.saltstack.com/topics/reactor/index.html ? is that right? because the documentation does not make it clear that or why multiple states in a reactor sls are not allowed...
20:54 MTecknology cachedout: except that I'm only managing ~10 users and it's pulling EVERY user on the network, if it's pulling that entire giant list, that's going to be horribly ugly
20:55 cachedout MTecknology: Ah. OK. That needs to be filed as a bug then.
20:55 fxhp http://pad.yohdah.com/203/salt-pillar-ordering-top | what causes ordering in top ?
20:56 kainswor I'm having strange issues getting highstate to work as expected. I can run state.sls with states in the base env, and can cp.get_file to push the top.sls file as expected. But when I run highstate I get "No top file or external nodes data matches found" (and this isn't true of the actual top.sls)
20:56 JasonSwindle basepi:  Any movement on this? https://github.com/saltstack/salt/issues/9587
20:56 MTecknology cachedout: it's there, wanna let me know if there's not enough detail? (after meeting time)
20:57 cachedout MTecknology: I just glanced at it. Could you please add a little bit about your particular auth setup (AD + nslcd) so we can have that info?
20:57 abhi09 joined #salt
20:59 kainswor For what it's worth- the setup I'm using was working perfectly with 0.15.3, and I only got the highstate errors after upgrading to 0.17.4
21:01 cewood joined #salt
21:03 Gifflen joined #salt
21:03 JasonSwindle kainswor:  I feel your pain.  I am going from 0.16.4 to 0.17.4
21:03 hunter joined #salt
21:03 kaptk2 joined #salt
21:04 MTecknology heh... one of the states turned nslcd back on and now the waiting game
21:04 cachedout Heh. Caught in your own web! ;]
21:05 bhosmer joined #salt
21:06 ajw0100 joined #salt
21:06 hunter I'm a new user of salt and making some progress. An early question for me (that I can't _quite_ find in teh docs, but I've looked) is how to have a single sls file (I've named go.sls) that installs multiple packages that our devs want for the go compiler.
21:07 JasonSwindle hunter:  Do they need to be ordered?
21:07 hunter I come from puppet, so I may have a puppet insprired mental block
21:07 JasonSwindle AKA, package 1 and then 2?
21:08 hunter JasonSwindle: no
21:08 JasonSwindle Ok, that is pretty easy.
21:08 jslatts joined #salt
21:08 JasonSwindle Let me show you an example.
21:08 JasonSwindle One sec
21:08 kainswor behavior: http://pastebin.com/e8v0aZdA
21:08 hunter Or url - I've got about 10 tabs opened to the docs, I just haven't found THAT answer yet.
21:09 scoates hunter: http://paste.roguecoders.com/p/e4dbed47379e254cda610269a5b17214.txt
21:09 dancat joined #salt
21:09 scoates where depN is a package name
21:09 JasonSwindle That works
21:09 JasonSwindle or something a lot more complex
21:09 JasonSwindle https://gist.github.com/JasonSwindle/36f2eb94a337b51697c6/raw/e03805020cbfcb6ed9bff517fab3e9311d11d1db/gistfile1.txt
21:10 kainswor hunter: http://docs.saltstack.com/ref/states/all/salt.states.pkg.html?highlight=pkg.installed#salt.states.pkg.installed did you find that
21:10 JasonSwindle That will install GIT, KIBANA, and NGINX
21:10 JasonSwindle Just copies some of my SLSs into one gist
21:10 * scoates nods
21:10 JasonSwindle but pkgs maybe better, if it is pretty cut and dry
21:11 hunter kainswor: no, but I don't think that answers my specific question (I'm going fast as apparently I'm reading from the irc firehose) :)
21:11 hunter scoates: that answers my question quite well
21:12 MTecknology cachedout: k- a bit more added
21:12 ekristen so pillar data doesn’t stack?
21:12 ekristen from different env?
21:12 cachedout MTecknology: Great! Thanks!
21:12 kainswor hunter: the example scoates gave is a usage of the module I linked ;) in the docs you can see more examples for specifying versions etc.
21:13 Gifflen joined #salt
21:13 scoates ekristen: I think it's supposed to, but I never managed to get it to work.
21:13 hunter kainswor: Oh I'm gonna dig into that - that page and the example from Jason leads me in interesting directions.
21:13 hunter But my exact question was answered by scoates paste.
21:14 JasonSwindle hunter:  I do not like using pkgs for personal reasons, but it works pretty well.
21:14 hunter ???
21:14 JasonSwindle if I am going to install something, I make it a package so I can reuse it in other project
21:14 ekristen scoates: yeah it seems that my dev pillar info replaces my base pillar
21:14 hunter How else do you insure a needed rpm is present ?
21:14 scoates ekristen: yes. I was hoping it would assign and override
21:15 JasonSwindle I just use pkg.installed without pkgs
21:15 hunter JasonSwindle: I suspect here that pkg and package are nomenclature I'm not yet familiar with
21:15 hunter As in the differences thereof.
21:15 zzzirk joined #salt
21:15 JasonSwindle I have a "packages" folder that I try to use on all of my projects
21:15 ekristen scoates: that was my understanding too, but doesn’t appear to be working that way
21:15 JasonSwindle trying to keep things dry
21:16 hunter JasonSwindle: what does that folder do?
21:16 tmwsiy can the windows package sls files take ftp:// prefixed url's?
21:16 JasonSwindle Just a folder. :)
21:16 JasonSwindle but in my top.sls I have something like this
21:16 hunter And whats contained in there? (Triggering yum or apt installs is going to be key to us - are you not using the system specific packages?)
21:17 JasonSwindle hunter:  https://gist.github.com/JasonSwindle/4f0d683da045be3d8b16
21:17 jacksontj joined #salt
21:17 JasonSwindle I am using system specific packages
21:18 JasonSwindle Updated the gist with an vim.sls from the packages folder
21:18 JasonSwindle This in no way is the *only* way to do salt
21:18 JasonSwindle it is just the way I did it
21:18 timoguin hunter, pkg.install will use the system package manager. so you just pass it the package name like in scoates example.
21:18 JasonSwindle To try to keep things in order and try to reuse things
21:19 JasonSwindle So for example, I am on Ubuntu…. and requesting Salt via apt to install 'vim-nox'
21:19 scoates we also build .deb + apt all of our non-distro packages because that makes life much easier.
21:19 scoates (so I can use pkg.installed for everything except our app git repos)
21:19 tmwsiy I have bigger issues. when I run pkg.installed or pkg.refreshdb against my minion it tells me that they are not available using 0.17.4 on both master and minion
21:20 hunter JasonSwindle: I think I see what you are doing. Package name and count differences between ubuntu and centos and soon to be solved on my list.
21:20 tmwsiy any idea the best way to start troubleshooting this?
21:20 JasonSwindle -l debug
21:20 JasonSwindle ?
21:21 hunter scoates: I wish for us- our devs have no idea
21:22 timoguin hunter, check out this SSH formula for a pretty good way to handle writing states that will work on multiple disto: https://github.com/saltstack-formulas/ssh-formula
21:22 timoguin there's a map.jinja file that defines the package names for each distro
21:22 timoguin and then the sls loops through that and uses the name for whichever distro it's running on
21:23 timoguin a lot of the formulas are using that convention now
21:23 JasonSwindle I agree, that is a really nice system
21:23 JasonSwindle I wish I had that when I started…. -_-
21:25 Ryan_Lane2 joined #salt
21:25 MTecknology cachedout: I'm just hoping I can fix this with config and not need to bump my entire infrastructure to 0.17.5
21:26 cachedout MTecknology: Aye, makes sense.
21:26 tmwsiy JasonSwindle: There is nothing reporting an error, just wondering why those states are no available.
21:26 JasonSwindle Does the minion log give anything?
21:27 JasonSwindle Maybe run salt-call on the minion with debuging?
21:27 JasonSwindle No idea, that is odd.
21:28 timoguin tmwsiy, how are you "running" them?
21:28 timoguin CLI, SLS files?
21:30 tmwsiy timoguin: through the cli with #salt 'win*' pkg.installed
21:31 timoguin tmwsiy, you need this: salt 'win*' state.single pkg.installed foo
21:31 timoguin otherwise it will be trying to execute the standard execution modules
21:32 timoguin and those are state modules
21:32 timoguin so, put them in an SLS file or use state.single to pass them straight on the command line
21:33 hunter timoguin: Why the jinja there?
21:33 tmwsiy timoguin: awesome thanks so much!
21:33 hunter Thats a poor question - let me try again.
21:33 kainswor okay I've got some perplexing results: when I replaced the python generated top file with a static sls highstate functions as intended
21:34 kainswor but I know the python generated file is good- it was working with v-.15.3
21:34 hunter timguin:  I understand line 1-11 - what teh devil does line 12 do?
21:34 kainswor what could cause a python top.sls not to render properly and cause 'state.show_top' to just draw a blank
21:35 KyleG left #salt
21:38 timoguin hunter, lol.... honestly i don't really know. it's still a bit of magic to me. i'm not great with jinja
21:38 Marion joined #salt
21:38 hunter fair enough - at least its magic for both of us - Thank You for that example.
21:41 jdenning hunter / timoguin: this looks in the pillar to see if there is a param 'ssh', and if so it will merge that dict, with the dict given by the filter_by function
21:41 hunter timoguin: it certainly does what I want except I'm not sure how to handle the case of "its one package on ubuntu and 2 on redhat"
21:41 jdenning (so you can override what is in this file with your pillar values)
21:41 Gifflen_ joined #salt
21:42 hunter Thats interesting - I haven't attacked pillar yet.
21:42 ajw0100 joined #salt
21:43 timoguin well i know you can do {% if os is redhat %} do this {% endif %}
21:43 timoguin except not that exact syntax
21:43 timoguin so worst case scenario you have an SLS with a couple of if blocks
21:44 hunter Yes I've seen that - in the main sls files. thats why I prefer this example. What I think I need is an example where the packages is a list
21:44 ekristen man salt-cloud maps are buggy as hell
21:44 ekristen salt-cloud -m dev.map — works just fine and shows me that my instances are already running
21:45 ekristen but if I pass -d it craps out saying that the hosts are invalid keys or something like that
21:45 timoguin hunter, yea we need some more examples. this kinda thing is fairly new for the formulas
21:45 hunter I hate to say it here but the puppet patterns for this are moderately well understood.
21:47 timoguin well yea, puppet IS more mature.
21:48 timoguin but i didn't say that
21:48 worstadmin joined #salt
21:48 jdenning hunter: You could do something like this: http://pastebin.com/UiP9UkT2
21:49 JasonSwindle abe_music: You here as well?
21:49 abe_music JasonSwindle: yup
21:49 jdenning (if you don't like jinja, there are alternate renderers, but jinja is the quickest out-of-the-box)
21:49 abe_music oh wait
21:49 kainswor Is the py renderer intended only for state sls files?
21:49 abe_music i'm here as in IRC
21:49 kainswor (as in not top.sls files)
21:49 JasonSwindle abe_music:  I see we are having somewhat the same issue on dev branch
21:49 timoguin I like jinja... I'm just not very good at it yet.
21:50 abe_music JasonSwindle: yeah, i gave techhat a revision that was working fine for me...it's a bit old though
21:50 JasonSwindle I know it works in 0.17.4…. if you want to go back THAT far.  lol
21:50 jdenning kainswor: I haven't actually tried it, but I'm fairly sure the renderers work for all the sls files
21:51 jdenning (including top.sls)
21:51 abe_music JasonSwindle: this is the revision i was using https://github.com/saltstack/salt/commit/f5810fdc79a6a51c06f20c631ee2ffe4ae7bd2ef
21:51 JasonSwindle I have not tried testing yet where it breaks.
21:51 NotreDev joined #salt
21:52 JasonSwindle Sounds like you guys are onto something
21:52 JasonSwindle I hope this makes it into RC2.
21:52 abe_music do you know when RC2 is being cut?
21:52 jdenning hunter: I just realized there's a syntax error in that pastebin, but you get the idea, right?
21:52 ekristen anyone know if you can define subnetid, securitygroupid in a profile with salt-cloud or does it have to be in the provider?
21:53 timoguin generally you want it to be in the provider, in case you ever want to add another one and not have to change your configs
21:53 david_a joined #salt
21:55 kainswor Hm- because I replaced a #!py top.sls with a quick on I wrote and my error went away {Result:    False         Comment:   No Top file or external nodes data matches found}
21:56 worstadmin joined #salt
21:56 jdenning kainswor: Can you post the code?
21:56 jdenning (i.e. to pastebin)
21:56 Ahlee external_auth and groups, possible?
21:57 Ahlee i want to grant everything to anybody in my AD groups, but it looks like only users are definable?
21:58 jdenning Ahlee: I think it only works at the user level
21:58 Nazzy *grumbles*
21:58 Nazzy that 0.17.1 protocol change is an ass
21:59 Ahlee jdenning: was afraid of that.
21:59 Ahlee guess I open a defect
21:59 hunter joined #salt
21:59 jdenning you might want to ask on the google group, or ping someone who actually works @ SaltStack , I'm not an expert on the acl system :)
22:00 kainswor jdenning: http://pastebin.com/rD0UrQCg There aren't any issues running the python on its own, or getting a valid structure out of it
22:01 Ahlee https://github.com/saltstack/salt/issues/7791?source=cc
22:02 Linz_ joined #salt
22:03 cdsrv joined #salt
22:04 yomilk joined #salt
22:04 SpeeR joined #salt
22:05 aleszoulek joined #salt
22:05 jdenning kainswor: Yeah, it looks good to me..let me check something
22:05 abe_music favorite vim shortcut of the day: zz
22:06 basepi JasonSwindle: redbeard2 says he's working on it now.
22:06 thrashr888 joined #salt
22:06 JasonSwindle basepi:  AWESOME!
22:06 jdenning kainswor: Yeah, it seems that the external renderers *are* just for state data - http://docs.saltstack.com/topics/development/modular_systems.html#renderers..guess you'll want to look at writing a custom tops module
22:06 basepi ya, he's actively working on the --parallel one that you commented in, and he thinks your fix will be right in line
22:07 JasonSwindle I hope. :)
22:07 jdenning kainswor: Actually, looking at what you are actually trying to do, a small tweak to the custom tops module I've been working on would do what you want
22:07 kainswor feeling generous? :)
22:08 darless left #salt
22:08 jdenning heh :) I actually was thinking about making an http-based tops module anyways
22:08 jdenning I'm planning on releasing a couple of salt-contrib modules this weekend, so if you can wait that long, sure :)
22:10 kainswor I'll definitely follow that- probably gonna scrap something together for now to get things working
22:10 jdenning I'll announce on the google group, or if you want to poke at me next week, I'm usually in this channel
22:12 ekristen how can I monitor any running jobs from salt master
22:14 Ahlee ekristen: salt-run jobs.active
22:16 ekristen hrm that throws an exception
22:16 Nazzy basepi, do you know of any incompatibilities between 2014.1/git and 0.17.4?  my pillar data is apparently invisible
22:16 ekristen KeyError: tgt
22:17 Katafalkas joined #salt
22:18 hunter joined #salt
22:18 MTecknology {% set centers = pillar.get('centers_served', [grains['id'].split('.')[1]]) %}  <-- this 'almost' works, except that it doesn't, If the id is server.0040.domain.tld, then it becomes 40 instead of 0040. If I add double quotes around that, then it uses that bit literally...
22:18 hunter jdenning: that was exactly what I meant. Thank You.
22:19 jdenning hunter: glad to help :)
22:19 MTecknology I tried pillar.get('centers_served', ["grains['id'].split('.')[1]"]) which makes that string be read literally, I tried pillar.get('centers_served', ['"'+grains['id'].split('.')[1]+'"']) which just makes it mad
22:20 basepi Nazzy: not specifically, but there has been a *lot* of churn since the 0.17 branch, so it wouldn't surprise me.
22:20 basepi go ahead and file an issue, Nazzy
22:21 MTecknology even 0.17.1 and 0.17.4 had a lot of compatibility issues that I saw
22:21 Nazzy basepi, can do ... was really hoping to bypass having to dig too deeply heh
22:22 basepi Nazzy: ya, sorry.  master is on the RC and minions are on 0.17.4?
22:22 viraptor1 joined #salt
22:22 jslatts joined #salt
22:22 Nazzy basepi, other way round ... master is 0.17.4 and minion is on gentoo's salt-9999
22:24 kainswor dummy question here: the master config has a runner_dirs option for specifying custom runners.. how would I specify a custom top module
22:25 Hefeweizen joined #salt
22:25 clearclaw Okay, I'm back to something between overstates and remote control states.
22:26 basepi Nazzy: generally we recommend running your master at least as high of a version as your minions, but i still think it probably should still work
22:26 basepi Nazzy: so create an issue, we'll see if we can see what
22:26 basepi what's going on
22:27 mohan__ left #salt
22:27 jdenning kainswor: you need to set master_tops in the /etc/salt/master config
22:28 taion809 joined #salt
22:28 db joined #salt
22:28 jdenning also, if you are trying to use a custom module (i.e. not included w/ salt), you also need to set extension_modules
22:29 kainswor okay so it should go in my extension_modules dir? thanks
22:29 Nazzy basepi, I'd love to, but doing tracking on ubuntu is probably way more effort than I want to tackle at this moment... I could run it from a virtual env version I was using for testing, but I suspect that will break the other nodes instead :/
22:29 jdenning well, you set the config value to the extension_modules dir, and then make a subdir for tops modules
22:29 Nazzy basepi, I'm writing the issue up now, depending on what response I get I'll start digging through the code tomorrow if I have to :)
22:29 jdenning e.g. /srv/salt/ext/tops
22:30 basepi Nazzy: sounds good, sorry you're running into troubles, this is why we release release candidates. =)
22:30 Nazzy basepi, pft, where would the fun be if there weren't any bugs? :p
22:31 kainswor thanks! sorry for the little questions- it's really helpful getting the answers vs. tripping around till I find it though :)
22:31 __number5__ Is there anyway to pass in parameter --no-use-wheel to pip in pip_state?
22:31 basepi Nazzy: ha!
22:31 jdenning no prob :)
22:31 __number5__ pip suddenly changed default format to wheel
22:32 ekristen how can I debug why my reactor isn’t firing?
22:33 bhosmer joined #salt
22:34 jdenning ektristen: I like to watch the 0MQ socket with another client to make sure the event messages are coming through as expected
22:34 lpmulligan joined #salt
22:35 jdenning ektristen: Check out https://github.com/saltstack/salt/blob/develop/tests/eventlisten.py
22:35 ekristen jdenning: I grepped my master log and I see “Gathering reactors for tag salt/auth”
22:35 ekristen jdenning: but I don’t see it for “salt/minion/ink*/start"
22:35 ekristen I’m using http://docs.saltstack.com/topics/reactor/index.html as starting point
22:35 ekristen I want new instances to run highstate when they come online for the first time
22:36 jdenning do you have logging set to debug on the master?
22:36 jdenning (log_level_logfile: debug)
22:36 ekristen yes
22:36 pmcg joined #salt
22:36 jdenning no relevant errors or anything?
22:37 ekristen oh
22:37 ekristen I see it
22:37 ekristen I missed that ink* is part of the id
22:37 ekristen I should do start/minion/*/start
22:37 jdenning ah, yep, that will make a difference :)
22:38 ekristen time to destroy my instances and try again ;)
22:39 EugeneKay joined #salt
22:42 Ahlee when targeting with env=Staging, I have gitfs configured with a Staging branch
22:42 Ahlee http://pastebin.com/wstzJQTv
22:42 * ekristen crosses fingers
22:42 Ahlee this has worked for freaking weeks.  merge a new item in
22:42 Ahlee and now it blows up
22:43 Ahlee deleting /var/cache/salt/master/gitfs, again. trying again
22:43 jdenning Ahlee: did you recently upgrade?  I saw someone mentioning a potential issue w/ the git_fs backend upon upgrading today
22:44 Ahlee jdenning: no
22:44 Ahlee 0.17.2
22:44 jdenning did you change your top data?
22:44 jdenning (I'm thinking you typoed 'Staging' or something?)
22:44 Ahlee should not have changed, no
22:45 Mua joined #salt
22:45 Ahlee i have multiple gitfs roots, i'm guessing it's freaking out over my attempt to move a file out of one gitfs remote into another
22:46 Ahlee and no, Staging is very meticulous, i've just verified both with the git server and locally on hte server that Staging branch exists in all remotes
22:46 Ahlee dropping to -l all
22:47 Nazzy basepi, opened as #9745
22:47 basepi Nazzy: thanks
22:49 jacksontj joined #salt
22:50 ekristen Ahlee: how do you do development on states when using a gitfs remote?
22:50 Ahlee ekristen: a lot of cursing
22:51 Ahlee in theory it's branch, develop in branch, target with env=branch, then merge in
22:51 ekristen right, so are you developing with a file:// gitfs locally on the salt-master on the dev branch?
22:51 Ahlee no.
22:52 Ahlee my servers are in environemnt=something
22:52 Ahlee when I want to edit something, i branch something to something_feature
22:52 Ahlee i then target one server directly with state.sls env=something_feature
22:52 Ahlee i then target a few servers, then merge in to something and target all
22:53 ekristen right so are you developing your states on you workstation then push to a remote git server and then just have the git server referenced in the salt master?
22:53 Ahlee yes
22:53 hunter Is there an alias feature? I have a client with very.long.name.foo that I'm deving against and getting tired of typing that all the time.
22:53 Ahlee sorry, i didn't get that
22:53 bhosmer joined #salt
22:53 ekristen ugh, so if you typo a jinja template you have to push again to your remote? ;(
22:53 Ahlee hunter: salt 'very*'
22:53 EugeneKay There's the bash autocomplete scripts, but they're fiddly
22:53 Ryan_Lane2 joined #salt
22:53 EugeneKay Or yeah, *
22:54 ekristen so gitfs_remove “branch_name” maps to salt environment?
22:54 Ahlee hunter: there's also PCRE matching
22:54 hunter There's no very=very.long.name ? (Hate * on the command line )
22:54 ekristen gitfs_remote*
22:54 EugeneKay You could set a Grain and match against that
22:54 ekristen Ahlee: salt environment by the same name I mean
22:54 ekristen I read about master = base, dev = dev, qa = qa, etc
22:54 hunter ekristen: oh? Where?
22:54 sylvinus joined #salt
22:54 Ahlee ekristen: yes, exactly. branches (and tags, but i never got tags working) match branches
22:55 Ahlee er, match environments
22:55 hunter (gitfs is on the salt todo list later)
22:55 Ahlee save yourself some pain and wait for the pygit2 implementation
22:55 ekristen what will that do Ahlee?
22:55 Ahlee pygit2?
22:55 ekristen yeah
22:55 Ahlee a lot of speed improvements and I'm hoping stability fixes as well.
22:56 srage_ joined #salt
22:57 ekristen I’m thinking I’m going to use the gitfs_remote: file://local/repo so I can do testing of new states from the salt-master
22:57 jdenning Ahlee: do you have your repos set up such that each app/state has it's own repo (like salt-formulas?)  I have a similar workflow, but only one repo
22:57 ekristen then commit them to the branch when they are working
22:58 jdenning but I've been wanting to bring things in line w/ the salt-formulas standard
22:58 Ahlee jdenning: i have a lot of legacy crap brought forward from before our migration from cvs -> svn -> git
22:58 Ahlee so, I have 5 remotes defined
22:58 jdenning ah, gotcha
22:58 Ahlee you need all remotes to have the same structure
22:59 Ahlee if one remote doesn't have env=Staging, for example
22:59 Ahlee it blows up
22:59 bhosmer joined #salt
22:59 Ahlee but, there's no real rhyme or reason to how the five are split
22:59 ekristen Ahlee: why? if it doesn’t have staging then it
22:59 Ahlee other than 10 years of organic growth all pulled into git, and no prioritzation to straighten
22:59 jdenning yeah, that's one of my issues with having the multiple repos - keeping the branches in sync..
22:59 ekristen doesn’t it just default to base/master
22:59 Ahlee ekristen: no, it throws exceptions.
22:59 clearclaw I have a standard(?) stack of entries in files: test, dev, qu, stage, prod etc like a stackable filesystem.  I was about to move that to GitFS.  Should I hold off?
22:59 Ahlee clearclaw: Personally I would wait.
22:59 jdenning I really want to be able to allow engineers to fork and build their own branches, so they can customize stacks for dev
23:00 clearclaw Ahlee: Why would you wait?
23:00 ekristen Ahlee: I have 3 gitfs_remotes right now, and 1 is an empty remote, the other two are saltstack-forumlas that do not have dev branches and it works just fine for me?
23:00 * clearclaw is newvous of all this config not backed up.
23:00 Ahlee clearclaw: becuase I know what's coming, and I've fought what's currently there.
23:00 Ahlee clearclaw: if i was starting right now, i would clone locally.
23:00 * clearclaw nods.  Unspecific "there be dragons".
23:00 jdenning clearclaw: Yeah, getting the GitFS setup is kind of a PITA..
23:01 Ahlee clearclaw: they've rewrote the git from GitPython to pygit2
23:01 Ahlee if you've used those libraries, you'll know how significant it is
23:01 Ahlee if you haven't, well, it's significant
23:01 jdenning basically, there were some issues depending on what version of gitpython you had installed..+1, gitpython sucks
23:02 steve1 joined #salt
23:02 Ahlee i hadn't used them until i heard they were rewriting, so i poked my head into both
23:02 Ahlee and, well.
23:02 * clearclaw sighs
23:02 Ahlee Dragons indeed.
23:03 Ahlee if you have a small number of minions, small number of files, gitfs works now
23:03 Ahlee either of those not true, you end up with a very sore forehead
23:04 clearclaw I have ~8 environments.  Each environment is small (handful of systems).  All in AWS.  All publicly routable.  I'm getting reading to move each environment to its own VPC, with the assumption that I'd use a syndicated Salt-master to bridge from the external/global salt master into each VPC.
23:04 clearclaw (8 is growing)
23:05 clearclaw My state trees are extensive, but not particularly complex (yet).
23:06 clearclaw Things will get messier when I figure out how to integrate South for DB schema migrations.
23:06 kainswor Okay- I've got my custom tops module working(ish). When I call salt target state.show_top I get what I expect, but when I call salt-call state.show_top on a minion I nothing
23:07 hunter I just attempted to add nodegroups via /etc/salt/master.d/nodegroups.conf
23:07 matanya joined #salt
23:07 hunter salt -G group_name complains - is there a way to get the server to dump what the nodegroups are so I can see whats going on?
23:08 hunter (I restarted the salt-master after adding the default include to the master config - so it _should_ be looking at that file)
23:08 ekristen jdenning: it seems that highstate was run looking at my minion logs, but as far as I can tell none of my states were run :/
23:09 erjohnso joined #salt
23:09 ekristen can highstate run on a minion that has authed but not get the states from the master?
23:09 hunter Nevermind - its not -G its -N
23:10 abhi09 joined #salt
23:10 Ahlee oh god damnit.
23:10 Ahlee a warning? seriously? 2014-01-14 17:09:41,006 [salt.loaded.int.fileserver.gitfs            ][WARNING ] GitPython exception caught while fetching: len([]) != len(['Permission denied (publickey).', ''])
23:10 clearclaw Having problems with Remote Control States/Overstates: http://pastebin.com/RCYNABdj
23:11 clearclaw Any ideas?
23:11 ekristen anyone using a reactor to initally tell your minion to run highstate when it first comes online? I can’t seem to get it to work
23:11 jfzhu_us joined #salt
23:11 clearclaw ekristen: I have that setup and working, but currently disabled.
23:11 Katafalkas joined #salt
23:11 ekristen hrm
23:11 clearclaw ekristen: I'll put together a pastebin for it...
23:13 clearclaw ekristen: have a look at this: http://pastebin.com/ATGRN3HJ
23:13 ekristen that is what I have
23:13 ekristen my minion says it ran highstate, but none of my states were run
23:14 clearclaw Ahh.  They get run for me.
23:14 ekristen so I was wondering if somehow my states are transferring?
23:14 iShming joined #salt
23:15 clearclaw Note that that reactor runs every time the minion comes up, such as from a reboot or service restart
23:15 jdenning ekristen: I was going to ask if you were using the cmd.state.highstate command, but I assume so (same as clearclaw's example)
23:15 iShming whiteinge: Guess who's writing states?
23:15 ekristen jdenning: checking now
23:15 abhi09 joined #salt
23:15 clearclaw Can anyone help with: http://pastebin.com/RCYNABdj
23:15 clearclaw ?
23:15 hunter salt -N france compilers.go env=dev
23:15 ekristen arg, well can’t cause I’m doing something else on my master at the moment, but I’m 99% sure I’m using cmd.state.hightstate
23:16 * whiteinge high-fives iShming!
23:16 hunter fails - I'm attempting to specify a single sls file wrong. the partial path is /srv/salt/dev/compilers/go.sls
23:16 hunter So how would i test that in a salt invocation? (I thought I understood this)
23:16 kainswor any reason a custom tops module wouldn't run when using "salt-call state.show_tops" from a minion
23:17 iShming whiteinge: Thanks.  :-)
23:17 Ahlee ekristen: here's wiat I have, which only runs sync_modules: http://pastebin.com/gyrsE5RL
23:17 Ahlee which is almost identical to hunter's
23:17 iShming so it's time I started chatting with youze guys...
23:17 jdenning ektristen: do the states run correctly if you just do `salt '*' state.highstate` ?
23:17 jdenning ekristen ^^
23:17 ekristen jdenning: yes
23:17 whiteinge iShming: nice you have you here :)
23:17 whiteinge s/you/to
23:17 seanz joined #salt
23:17 zzzirk joined #salt
23:18 iShming I am not a good IRC surfer so I'll try to hang with the crowd...
23:18 seanz iShming: Greetings!
23:18 jdenning ekristen - strange..when you can access your master again, could you post your /etc/salt/master.d/reactor.conf (or whatever relevant config file section), and the reactor sls file you are using?
23:18 iShming I'm more of a "blurt some misformed question that nobody answers and then leave" ircer.
23:18 iShming Well seanz we meet again!
23:19 whiteinge haha
23:19 ekristen jdenning: I basically copied and pasted the stuff from the reactor man page
23:19 seanz whiteinge: Greetings to you as well.
23:19 ekristen but yeah I will
23:19 seanz A small world, filled with strangers.
23:19 seanz IRC is probably the best example of what logging looks like when multiple threads log to the same file that I can think of (at the moment)
23:19 jdenning kainsor: maybe try deleting the cache on the minion and restarting?
23:20 whiteinge at least until the cold kills us all. or possibly the heat
23:20 jdenning kainswor  ^^
23:21 seanz whiteinge: Do you know something about this upcoming apocalypse, or am I just barging in on an existing conversation?
23:21 ekristen jdenning: https://gist.github.com/ekristen/c3989e1620d4b7488335
23:22 tyler-baker joined #salt
23:22 tyler-baker joined #salt
23:22 iShming ambiguous "right".
23:22 whiteinge between reading about the cold in the mid-west or the heat in australia i figure one of 'em is about to take us out
23:23 whiteinge although if i was in australia i'd be equally concerned about the spiders
23:23 iShming Alright.  So the question is....  (heat in Australia?)
23:23 Gifflen joined #salt
23:23 jdenning ekristen: yep, that all looks good to me
23:24 kainswor jdenning: no dice :/
23:24 jdenning ekristen: Maybe make an additional state (like touch a file on the master) so you can verify the highstate is running?
23:25 ekristen not sure I follow
23:25 jdenning I'll add a comment to the pastebin, one sec..
23:26 matanya joined #salt
23:26 troyready joined #salt
23:26 iShming I've got a pillar sls file with this line:  {% set code_dir = salt['pillar.get']('path:to:the:info') %}
23:27 kermit1 joined #salt
23:27 iShming But it seems to not have that information, even though it's there when I do salt-call pillar.items it is there (Self-mastered minion)
23:28 iShming whiteinge: seanz said you might have some knowledge of such matters?
23:28 Ryan_Lane2 joined #salt
23:28 jdenning ekristen: https://gist.github.com/ekristen/c3989e1620d4b7488335
23:28 jdenning (oops, indent got messed up for some reason)
23:29 ekristen no worries
23:29 whiteinge iShming: you can't get pillar from pillar :-/
23:29 ekristen I get the picture
23:29 iShming I was afraid you'd say that....  whiteinge
23:29 iShming But you can get grains in pillar then.
23:29 whiteinge iShming: another tack could be to define an "include" file of sorts that you can fetch from multiple places in pillar
23:30 whiteinge or grains work too, yes
23:30 iShming Yeah, working on the includes...
23:30 iShming Thanks!
23:30 whiteinge salt can import json or yaml files natively
23:31 whiteinge so you could put variables into one of those
23:31 Ryan_Lane2 joined #salt
23:31 whiteinge iShming: http://docs.saltstack.com/ref/renderers/all/salt.renderers.jinja.html#template-serializers
23:32 mlister2006 joined #salt
23:32 iShming Nice...  heh.  I'm sure the ops team will love it when I convert everything to json.  :-)
23:32 ekristen jdenning: bbiaf, I’ll test it out and let you know how it goes
23:33 jdenning cool, good luck :)
23:33 iShming So that's an "import" instead of an "include"
23:34 ekristen I think it is time to upgrade my saltmaster to a larger ec2 instance
23:34 iShming Insane.
23:38 Ryan_Lane2 joined #salt
23:39 alunduil joined #salt
23:39 kainswor do master_tops not get sent to minions? I don't see any evidence that the tops get executed on a minion call to state.show_top
23:40 scott_w joined #salt
23:40 jdenning kainswor: sorry, I've never tried it from the minion side :(
23:41 abhi09 joined #salt
23:42 kainswor hmm- i might drop a ticket and try to resolve it. it doesn't make sense to have dynamic top content that can only be used by masters. wouldn't this be an issue for ext_nodes, etc.
23:46 abhi09 joined #salt
23:54 jfzhu_us1 joined #salt
23:54 kainswor I'm full of issues today: getting exceptions from the minion running highstate: http://pastebin.com/FTfBNmhE
23:56 jeddi kainswor: need statefile, not just error. :)
23:56 jdenning kainswor: trying to do `state.highstate test=true` ?
23:56 Ryan_Lane2 joined #salt
23:57 kainswor yes. at first. then I took out the test=true because I thought that might oddly be related and same
23:57 jdenning are you sure you erased the entirety of test=true (i.e. no lingering 'true' at the end)?
23:57 kainswor positive
23:58 jdenning that's a strange one then..
23:58 jdenning what's your exact command you are trying?
23:58 kainswor salt <target> state.highstate
23:59 jdenning try salt target test.ping - any errors?
23:59 kainswor all good

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary