Perl 6 - the future is here, just unevenly distributed

IRC log for #salt, 2018-03-28

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:27 sybix joined #salt
00:52 schemanic joined #salt
00:52 schemanic Good evening
01:04 Guest73 joined #salt
01:06 darix joined #salt
01:42 systemdave joined #salt
01:56 ilbot3 joined #salt
01:56 Topic for #salt is now Welcome to #salt! <+> Latest Versions: 2016.11.9, 2017.7.4 <+> RC for 2018.3.0 is out, please test it! <+> Support: https://www.saltstack.com/support/ <+> Logs: http://irclog.perlgeek.de/salt/ <+> Paste: https://gist.github.com/ <+> See also: #salt-devel, #salt-offtopic, and https://saltstackcommunity.herokuapp.com (for slack) <+> We are volunteers and may not have immediate answers
02:03 schemanic Hi, I'm having minion did not return issues again and I cant' get my machines to run highstate. it's like they don't even try
02:03 hemebond schemanic: Like they don't even try?
02:03 hemebond Have why you checked the minion logs?
02:03 hemebond Do they highstate when using salt-call?
02:05 schemanic calling highstate is just state.apply now right, not state.highstate yes?
02:05 hemebond Correct.
02:05 schemanic this is after I initially commission the server with salt-cloud
02:06 schemanic gtmanfred said something about when they do this it means the minion didn't even see the job
02:06 schemanic I can walk down the list of states and they each apply individually
02:06 hemebond Are you highstating manually?
02:07 schemanic yes
02:07 schemanic I'm calling salt 'my_minion' state.apply
02:07 schemanic from the master
02:07 hemebond And have you checked the minion log?
02:08 schemanic where, on the master or on the minion?
02:08 hemebond On the minion.
02:08 hemebond /var/log/salt/minion
02:13 schemanic it's hard to parse out since I've been running states one by one trying to figure it out. I've run an equivalent of a highstate currently. I'm going to commission a new server and call highstate on that
02:18 schemanic I'm calling highstate on the new minion. I should be able to get a log entry from it soon
02:20 schemanic hemebond: the log isn't present
02:20 schemanic part of the highstate mounts a new location for /var/log
02:22 schemanic hemebond, is it possible that thats what's giving me trouble?
02:30 tiwula joined #salt
02:36 hemebond It's possible.
02:36 hemebond Is the minion service running?
02:37 hemebond Can you run the service manually, in the foreground?
02:38 schemanic yeah, I can. I can stop and start it fine
02:39 schemanic calling highstate a second time works
02:39 schemanic the first time it does not though
02:39 schemanic the minion service was definitely running
02:40 hemebond Well you should have a log.
02:40 schemanic well now I do yes
02:40 schemanic the issue is that it wasn't there before
02:40 hemebond So if you have no log then something is affecting the minion on first run.
02:40 hemebond Are you remounting /var/log/ after the minion starts?
02:40 schemanic yes
02:41 schemanic i'm using a formula to mount and format a second drive for logs
02:41 schemanic so I'll provision the server, call salt 'my-minion' state.apply mounts
02:41 hemebond Right. Then are you restarting all services that have already started using the existing log directory?
02:41 schemanic no
02:42 schemanic I'm expecting salt to just run highstate all at once
02:42 hemebond So other services might be losing logs too?
02:43 schemanic possibly yes. I'm making the mounts formula the second or third thing that runs in my topfile. My assumption was that salt would run that first, but if salt gets stuck if it can't write to it's own log after a remount that might explain what's going on
02:51 exarkun joined #salt
03:05 copec joined #salt
03:16 evle joined #salt
03:16 alvinstarr joined #salt
03:32 zerocoolback joined #salt
04:06 hop joined #salt
04:08 emdk joined #salt
04:18 emdk left #salt
04:20 Its12am joined #salt
04:30 aruns joined #salt
04:38 Guest73 joined #salt
04:45 Guest73 joined #salt
04:46 zerocoolback joined #salt
04:59 Hybrid joined #salt
05:10 zerocoolback joined #salt
05:18 sauvin joined #salt
06:12 exarkun joined #salt
06:16 jas02 joined #salt
06:17 indistylo joined #salt
06:18 jas02 joined #salt
06:22 inetpro joined #salt
06:39 inetpro joined #salt
06:43 aldevar joined #salt
06:51 Tucky joined #salt
06:51 inetpro joined #salt
06:55 JAuz joined #salt
07:03 tyx joined #salt
07:07 Guest73 joined #salt
07:08 indistylo joined #salt
07:10 __peke___ joined #salt
07:10 aruns joined #salt
07:19 aviau joined #salt
07:19 DanyC joined #salt
07:23 Sketch joined #salt
07:26 Sketch joined #salt
07:42 inetpro joined #salt
07:46 mechleg joined #salt
07:51 jas02 joined #salt
07:52 shpoont joined #salt
07:58 Ricardo1000 joined #salt
08:17 fringe joined #salt
08:22 sauvin_ joined #salt
08:24 sauvin_ joined #salt
08:26 fringe hi, salt-minion uses its own embedded python interpreter or OS one located in /usr/bin?
08:27 hemebond /usr/bin/python /usr/bin/salt-minion
08:30 pf_moore joined #salt
08:33 Mattch joined #salt
08:35 fringe or actually question was rather "if I use #!py in salt, which interpreter does it take and how do I change that?"
08:36 fringe I was not precise, sorry.
08:36 Guest73 joined #salt
08:42 inad922 joined #salt
08:51 indistylo joined #salt
09:09 hemebond I would assume it uses the same interpreter.
09:38 aruns__ joined #salt
09:47 Hybrid joined #salt
09:53 miruoy joined #salt
09:56 babilen I need additional jinja_filters/utils while rendering pillar top.sls - I know how to add the former once 2018.3 releases, but how do I go about the latter in the interim?
10:04 indistylo joined #salt
10:17 aldevar joined #salt
10:26 mahafyi joined #salt
10:31 DanyC joined #salt
10:36 Guest73 joined #salt
11:04 edrocks joined #salt
11:09 indistylo joined #salt
11:13 aldevar joined #salt
11:24 inad923 joined #salt
11:38 schemanic joined #salt
11:40 emdk joined #salt
11:48 emdk Really weird issues guys and I can't figure it out.  I'm running 2017.7.4 and have a simple state that applies UTC time to a handful of Ubuntu servers.  The state applies and I can verify (timezone.get_zone) but everytime I run the highstate, it says it needs to be set
11:48 averell joined #salt
11:55 averell joined #salt
11:55 inad923 joined #salt
11:59 andi- joined #salt
12:11 babilen emdk: Mind sharing your state and the output you refer to on one of http://paste.debian.net, https://gist.github.com, http://sprunge.us, … ?
12:12 Felgar joined #salt
12:13 emdk http://paste.debian.net/1017218/
12:15 emdk http://paste.debian.net/1017219/
12:17 andi- joined #salt
12:17 babilen ta
12:19 babilen What does timezone.get_hwclock give you?
12:21 emdk localtime for all
12:21 babilen So .. it isn't setting the hwclock to utc
12:22 babilen Could you try two states? One for UTC and one for the timezone?
12:23 emdk Ah ha!
12:23 babilen Which distro is that?
12:24 babilen Or rather: What's the value of the 'os_family' grain?
12:25 schemanic joined #salt
12:25 emdk Those servers are Ubuntu 16.04 LTS.  All my other servers are Debian 7/8
12:25 babilen Right
12:25 babilen Could you check the content of /etc/default/rcS and /etc/adjtime respectively?
12:26 Nahual joined #salt
12:26 schemanic Hello. Two questions: why would my minions not gather the fqdn_ip4 grain?, and Does the salt minion barf if it cant write to it's logfile?
12:27 babilen emdk: It looks as if the module is making changes to /etc/default/rcS while you'd want LOCAL or UTC in /etc/adjtime for releases after squeeze
12:28 inad923 joined #salt
12:30 emdk 0.0 0 0.0
12:30 emdk 0
12:30 emdk LOCAL
12:30 babilen emdk: Lets keep it in here
12:31 babilen Okay, I guess this is a problem with the current timezone modules, in that they assume that newer Debian releases still use /etc/default/rcS
12:31 babilen (while they should use /etc/adjtime)
12:41 emdk Settings /etc/adjtime from 'LOCAL' to 'UTC' cleared up the highstate output
12:42 emdk http://paste.debian.net/1017226/
12:46 emdk Thanks babilen!  I'll file a bug report
12:49 babilen ta, feel free to highlight me. The code in question is https://github.com/saltstack/salt/blob/develop/salt/modules/timezone.py#L540-L544
12:52 aruns joined #salt
13:00 aruns__ joined #salt
13:01 emdk Wow, you did a code dive :D
13:07 gh34 joined #salt
13:08 elektrix joined #salt
13:23 inad923 joined #salt
13:26 brokensyntax joined #salt
13:29 darix joined #salt
13:40 schemanic joined #salt
13:42 cgiroua joined #salt
13:42 Bico_Fino Hi, I created a custom runner, I can run it with salt-run fine, but when I try to assign the result to the variable I get a KeyError, here is the full error: https://gist.github.com/bicofino/2d12c89f04900e45669f44ae4f6a1c3a
13:44 Neighbour the returned data of a runner is a dict with the minion_id as key
13:44 Neighbour which you can see on line #2 :)
13:46 pcn Wait, you can invoke a runner in a minion state?
13:46 Bico_Fino Thanks, going try it.
13:46 Neighbour yes, but it gets ugly fast....also there's a big with nested runners (you can start an orchestrate runner from within a state run as an orchestration) etc)
13:46 Neighbour bug*
13:48 Neighbour I use the pillar.show_pillar runner a lot to get the pillar of a minion before it actually exists (based on what it would get from the pillar top.sls)
13:49 Bico_Fino Neighbour {% set token = salt['saltutil.runner']('minion.token')['minion01'] %} should do it right?
13:49 Neighbour pcn: ah, i think i spoke too soon...you can't indeed run a runner on a minion, since runners are, per definition, master-only
13:50 Neighbour Bico_Fino: You can try it...but now that I think more on it, I'm more and more drawn to think it won't work
13:51 Neighbour since runners only run on the salt-master locally
13:52 Neighbour Bico_Fino: Also, have you synched the custom runner to the minion already? (saltutils.sync_all)
13:52 Bico_Fino Neighbour yes.
13:52 Neighbour because the error suggests that the minion can't find it
14:02 schemanic_ joined #salt
14:05 schemanic_ joined #salt
14:05 zerocool_ joined #salt
14:06 evle joined #salt
14:07 msmith joined #salt
14:09 msmith https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.saltutil.html#salt.modules.saltutil.runner says it must be run on a master. is `minion1` the master's own minion?
14:11 Bico_Fino no
14:11 Bico_Fino Yeah, can't run on minion. Going use another approach.
14:12 msmith there are lots of other custom modules you can run on a minion, just not a runner :)
14:12 msmith well, with the particular special case exception of masterless i believe
14:12 Neighbour :)
14:12 pcn What are you trying to get?
14:13 Bico_Fino pass data safely from one minion to another. :)
14:14 msmith as in, securely, rather than via mine?
14:14 Bico_Fino mine don't work for this because I don't know which minion will use the data. :/
14:14 Bico_Fino I think I will use vault
14:15 Bico_Fino send the data to vault and use it as pillar
14:15 msmith mine makes it available for any other minion to read
14:15 msmith how often will this data change?
14:15 Bico_Fino pretty often
14:15 Bico_Fino like 5 min
14:15 msmith then pillar is going to be a bad choice, unless you also plan to resync the pillar each time
14:16 Bico_Fino Yes, going need to resync pillar every time before use it.
14:17 msmith something concerns me about how much load that might cause, unless it's only going to be one minion once every 5 mins
14:17 Bico_Fino It's an adhoc state
14:17 mchlumsky joined #salt
14:17 Bico_Fino so not going run everytime
14:18 msmith then perhaps a custom grain might be more appropriate?
14:18 msmith i'm guessing here since i don't know your use case
14:18 pcn It's weird right?  Salt has a bus, but it's not really a peer-to-peer bus.
14:18 Bico_Fino But I'm going need pass the custom grain from one node to another
14:18 msmith like i said, i don't know your use case
14:19 Bico_Fino Well, I think I found a way, not pretty but is a way. :)
14:20 racooper joined #salt
14:20 Bico_Fino grab the grain from the minion from master and save as an yaml and use the yaml on the state. :)
14:20 pcn Could you use reactors to have the minion put a request on the event bus for some info, and have the master run a state with the response invoking salt-call with a custom piller that contains the data
14:21 msmith i was going to suggest an event, with the custom data as key
14:21 Bico_Fino I tried to use reactor before
14:22 Bico_Fino but got too big imo
14:22 Bico_Fino Would be cool to salt.mine have better permissions
14:22 Bico_Fino like dynamic
14:22 msmith but it still depends on what exactly is consuming this information. it's rare that minions do something that's not directly controlled by the master so anything that says minion-to-minion is unusual
14:22 Neighbour i have a custom branch for that, but it's still a WIP :)
14:23 Bico_Fino Neighbour will work like external pillars?
14:23 msmith if you can wait for 2018.3 then you may find that slots will give you what you're looking for
14:23 msmith not sure tho, since this is more in orchestrate territory
14:24 Bico_Fino I will wait. No rush!
14:24 msmith so out of curiosity, what is it that needs passing between minions?
14:25 Bico_Fino a token
14:25 Bico_Fino that can be used to join a cluster
14:25 Neighbour Bico_Fino: It allows you to add minion targeting (with tgt and tgt_type, like in saltmod.function) in mine.send and mine.update allowing only the targeted minions access to that specific mine data
14:26 Bico_Fino Neighbour that will be awesome!
14:26 J0hnSteel joined #salt
14:27 zer0def msmith: methinks you could leverage `http` in reactor's jinja, then pass the result through to the minion
14:27 msmith i'm not actually sure that salt is the best way to pass that around. perhaps sdb might be closer but otherwise i'd probably be looking at using salt-api or pepper to write a script to fetch it from one minion and call another
14:27 Neighbour i'll have to checkout slots first...if it fulfills the same functionality, my branch is no longer needed (and, after extensive reviewing, would only be added to fluorine or later)
14:28 inad923 joined #salt
14:28 Bico_Fino msmith Thinking about using Vault
14:29 zer0def pcn: ^
14:29 msmith if these are short-lived or one-shot tokens then i wouldn't even store them anywhere. that's why i suggested a script
14:29 msmith zer0def: the jinja would execute at render time, so i'm not sure it would re-render when the event was triggered
14:30 zer0def msmith: so the resulting yaml isn't rendered every time the reactor is ran?
14:31 zer0def i think it is, at least according to the aws example i've written up
14:32 msmith haven't used it, it could well be, but i wasn't going to recommend something that may not be a documented feature :)
14:33 onslack joined #salt
14:33 onslack <gtmanfred> fixed
14:34 zer0def msmith: i see; well, for the record, https://github.com/zer0def/salt-reactive-aws-in-15/blob/master/salt/reactor/sqs.sls works as it appears, meaning key's removed when getting a termination event and orchestrate is ran when launch event is received
14:34 msmith good to know
14:36 msmith however in this case it appears like the token in question is only intended for use by one minion at a time, or perhaps only a restricted certain few
14:36 msmith certainly if it's only an adhoc operation then i'm still of the opinion that it doesn't belong in a state
14:38 schemanic joined #salt
14:39 zer0def yeah, might not necessitate a state run from a reactor, was basically remarking on not needing to rely on a minion putting events for the master (unless that's the original source to begin with)
14:42 msmith the token comes from a minion, if that's what you mean
14:43 zer0def ok then
14:45 msmith see now i'm just feeling evil. trigger an orch state that calls the cluster minion to get the token and post it as an event along with a custom pillar to indicate the new minion to be joined to the cluster, and then reactor sends that data out in another orch state to actually perform the join operation
14:45 J0hnSteel joined #salt
14:46 msmith if you wanted it to be completely saltified then the new minion would have to initiate this process, and either somehow wait for it, or continue the highstate as a result of the 2nd orch state
14:47 msmith then you're asserting the state you want - to be a member of the cluster - and the means to achieve that state
14:47 Bico_Fino you can't run a cmd.run to another minion inside a state?
14:47 Bico_Fino right?
14:48 msmith no. a state runs on a single minion, it can't interact with other minions
14:48 Bico_Fino even if you run from master, correct?
14:48 msmith orch can run things on minions as that's running on the master
14:49 msmith i'm not aware of anything in a minion state that allows it to target another minion. with the exception of mine data, but that's data rather than execution
14:51 zer0def yeah, in that case you'd rather send an event and have the master react to it… or just find a way to skip this round-trip somehow in the first place
14:52 msmith securely, tho? that's why i suggested a script in the first place, only way to control both sides of the issue
14:52 msmith i suspect the OP has gone now :)
14:53 msmith Bico_Fino: does any of this rambling help you in the slightest? :)
14:53 zer0def well, which components do we consider secure? if a master is compromised, you're absolutely wrecked, so i figure involving the master is a sane assumption in terms of security
14:54 Bico_Fino oh
14:54 Bico_Fino I'm here. haha
14:54 Bico_Fino msmith yes.
14:55 Bico_Fino I think best way, master send a command to minion and save the token into vault and share it via pillar.
14:55 GrisKo joined #salt
14:56 Bico_Fino since the token has a TTL, I think is fine to let it in the pillar.
14:56 msmith what's the sensitivity of this token?
14:56 Bico_Fino pretty high. :P
14:57 GMsoft joined #salt
14:57 zer0def Bico_Fino: could you write up what you're actually trying to accomplish? i'm unable to piece together what you're trying to accomplish from the backlog
14:57 zer0def sorry, if that means you're asked to repeat yourself
14:57 Bico_Fino zer0def sure, give me 5 min.
14:57 msmith new minion wants to join an existing cluster. needs token from a different minion to do it
14:57 zer0def thanks
14:57 GMsoft hi
14:58 Bico_Fino I will update the gist that I shared.
14:59 GMsoft mhh I'm running into a weird issue. it seems that gitfs isn't updating the local files
14:59 GMsoft according to the logs, it says that the repository is up to date
14:59 GMsoft but looking into /var/cache/salt/master/gitfs/refs/, I see the old file
15:00 GMsoft while the FETCH_HEAD git thing points to the latest commit as expected
15:00 J0hnSteel joined #salt
15:03 schemanic joined #salt
15:03 zer0def msmith: i think it would validate a state.sls call, but only because i find snooping on all job finishing events on the bus as potentially taxing
15:04 GMsoft I tried to remove the whole cache but it did not help
15:04 tiwula joined #salt
15:04 Bico_Fino zer0def https://gist.github.com/bicofino/2d12c89f04900e45669f44ae4f6a1c3a
15:06 J0hnSteel joined #salt
15:06 zer0def oh, so you're already running a state against minion01
15:07 zer0def actually, nevermind, i was confusing the two. ;)
15:07 Bico_Fino :P
15:08 msmith the concern i'd have is that this token will persist beyond the lifetime of the actual join request, ttl or not
15:08 tyx joined #salt
15:08 zer0def well, as soon as any job containing the token is sent, it's cached
15:13 Bico_Fino clear cache after the job run?
15:13 zer0def sure sounds like a good case for thorium, all i can come up with is three roundtrips and stashing the token in outside storage - one for when minion01 reports it has finished installing, one when cluster01 stashes the token and sends an event it has done so, one to react on minion01 which would then pull the token from the stash
15:13 zer0def sure, you *could* do that, but then you're flushing all other jobs you might want to audit at some point
15:14 zer0def if you want to not have an intermediate stash, you'd need to obfuscate it so that only the targeted minion is able to read it
15:15 J0hnSteel joined #salt
15:15 cewood joined #salt
15:16 inad923 joined #salt
15:17 zer0def in which case, i guess each machine could have it's own unique gpg key and all minions could be able to refer to a keyserver; i'm still considering whether kerberization couldn't fit the bill
15:17 GMsoft weird thing is that if I copy the file to another name in the same repo, the new file works
15:18 zer0def but then, you're just swapping out the token for a keytab
15:18 Bico_Fino yeap
15:21 zer0def i mean once the data is in the intermediate stash (i think you were considering vault?), it really doesn't matter how it's exposed, as long as it is
15:21 zer0def "how it's exposed" meaning the manner in which minion01 is capable of reading it
15:23 zer0def what kind of security threats are you concerned about, Bico_Fino?
15:24 Cadmus So there's a minor YAML featire that just bit me on the arse >_<
15:25 Bico_Fino zer0def someone grabs the token and mess with the cluster.
15:25 zer0def if your master is compromised, you're already smoked, so the only other threat would be for the token to overstay it's welcome, in which case passing ciphertext containing the token through events would be probably best
15:25 zer0def and for creating the ciphertext, i guess you could set up your own minion keyserver and leverage gpg
15:26 J0hnSteel joined #salt
15:26 msmith my concern would be something being able to get to the token at some point in the future because something gets changed and the person changing it doesn't realise the consequences
15:26 zer0def (bear in mind that i might be overthinking it)
15:27 msmith you might be able to make it perfectly secure right now by thinking of every potential access point, but can you also guarantee that those won't change in the future and inadvertently expose the token?
15:28 zer0def well, if a token has a TTL, it would imply it expires at some point, i'd think
15:28 inad923 joined #salt
15:28 msmith sure, but if it's generated in advance in order to be available for use, then there's always a usable token being stored somewhere
15:29 msmith that's what i'd avoid
15:30 zer0def well, i assumed the token is generated on the spot when master reacts on cluster01 to the event from minion01
15:32 msmith that's closer to how i'd implement this, yes. but not the approach that Bico_Fino has been describing
15:33 zer0def thinking about time-based tokens, but then you're concerning yourself with distributing seed… either way, there's a manner in which a secret and be leaked
15:34 zer0def can be leaked*, that is
15:34 J0hnSteel joined #salt
15:36 Bico_Fino I think can be avoided with low TTL token?
15:36 zer0def ideally, $token is one-time
15:36 zer0def meaning once it's used, it's worthless
15:38 Bico_Fino that would do t
15:39 zer0def either that or find some way to implement axolotl or signal
15:40 zer0def actually, signal, since it's a superset of axolotl
15:40 msmith even with a low ttl token, if it can be used multiple times and it's stored _anywhere_ then it's vulnerable to some small extent
15:41 J0hnSteel joined #salt
15:41 msmith i wouldn't be allowed to go into production with that risk
15:42 Bico_Fino can't pass the data without being cached?
15:42 zer0def well, if the token has a cryptographically secure ratcheting value associated with it, it wouldn't matter
15:42 MTecknology What's the tl;dr for token stuff?
15:42 msmith you _can_ pass the data if you control where it's received and sent
15:42 zer0def since once a this token has been used, it's already ratcheted forward in difficult to predict ways
15:43 msmith [15:57] <msmith> new minion wants to join an existing cluster. needs token from a different minion to do it
15:43 msmith MTecknology ^^
15:43 MTecknology heh, odd- but easily doable
15:43 zer0def but ideally, $token should be single use, spares you the trouble of overthinking it
15:43 KyleG joined #salt
15:43 KyleG joined #salt
15:44 * msmith sits back and lets the expert explain it ;)
15:44 msmith my solution is a pepper script
15:45 MTecknology minion_1: redis.setex(token, time, value), minion_2: salt.register(token), master: if redis.del(token): salt.accept(minion_2)
15:45 zer0def yeah, that's easy mode
15:47 MTecknology with not much extra creativity, you could easily make that one-time-use time-based tokens
15:47 msmith still leaves a key around tho, right? that's no different to the sdb/yaml/pillar suggestiosn
15:47 MTecknology why would there be a key left around?
15:47 msmith stored in redis
15:47 MTecknology for the duration of it's validity..
15:47 zer0def master deletes it in MTecknology's case
15:48 msmith the trigger for this would need to be an orch state in that case
15:48 zer0def that token has to go through some medium, be it stashed in vault/redis or as ciphertext in event exchanges
15:48 MTecknology you could do the whole thing simply in a reactor
15:49 zer0def the idea so far was that $token has a timed lifespan, not per-use
15:49 MTecknology you could easily stick a private key as a value into redis and pass a private key to the minion, then wait to delete until after validation, but then you have a tiny race
15:50 msmith what's the accessibility to a key in redis?
15:50 zer0def i'm glad each of us basically came to the same conclusion, just with different sets of means ;)
15:51 Bico_Fino Could use vault instead of redis
15:51 zer0def let's stop using specific software terms (redis, vault) and just call it $stash, since it really doesn't matter that much :P
15:52 Bico_Fino You have a point
15:52 MTecknology I like redis for otp because of how redis.del(foo) works
15:52 zer0def ideally the solution would be for the token to be one-time use and pray you don't have a persistent mitmer ;P
15:53 msmith i've not used any of them, that's why my solution was a pepper script to fetch the token from $cluster, store it only in a variable long enough to call $minion and have it join the cluster. but if that token has a lifespan in the cluster then anything able to see the job cache will see that token and be able to re-use it
15:53 msmith one-time tokens are the only way to avoid replay
15:54 MTecknology I don't know that I understand what the actual goal is or if this is just some thought experiment
15:54 msmith afaik it's a real case
15:56 MTecknology if redis.del(foo): auth_success() \n else: auth_failed(). I can't think of a possible race condition. Heck, you could even go extra super overboard and... <...>
15:57 dhwt joined #salt
15:57 zer0def well, actually, having a persistent mitmer isn't a case of prayer, but rather proper opsec, which is something you can blame a human for
15:57 msmith always assume it's a human that's going to mess up your carefully laid plans ;)
15:58 zer0def hasn't this been proven an infinite+1 number of times already?
15:58 msmith it's a universal fact, from opsec to the death star :)
15:59 DanyC joined #salt
16:05 MTecknology minion_1: pub, prv = get_time_pair(); redis.setex(minion_id, max_ttl, pub); salt.init(minion_id, prv);  minion_2: token = time_tok(prv); salt.auth(token);  master: pub = redis.get(minion_id); redis.del(minion_id); if pub and validate(pub, token): success(); else: fail()
16:06 MTecknology /that/ one does give you a tiny race condition if you're being mitm'd; my first one doesn't have that
16:07 zer0def so basically timed keypairs
16:08 zer0def already brought up with gpg ;P
16:08 MTecknology I don't really think the time based OTP makes sense..
16:09 zer0def if it's time-based, it's not otp
16:09 MTecknology You're not reading what I wrote then?
16:09 MTecknology I never gave any solution that wasn't otp
16:09 MTecknology this one is just also time-based
16:10 MTecknology and it's also a maximum of one attempt, except for that tiny race potential
16:10 MTecknology in between the .get() and .del()
16:11 MTecknology You really shouldn't have to worry about a MitM since you can include the salt master's pub key in a template.
16:14 zer0def actually between .setex() and .del()
16:14 rivyn any of you guys know if VIM has a YAML+Jinja syntax?  Or just the yaml one?
16:14 zer0def well, the race is between. get() and del(), the opportunity to read is much longer
16:15 zer0def but yeah, this is just gymnastics at this point
16:16 zer0def rivyn: built-in? i think only yaml, but there's a vim script on saltstack's github
16:16 rivyn ahh, https://github.com/saltstack/salt-vim :)
16:17 MTecknology zer0def: ehm... is your token storage untrusted?
16:17 zer0def tbh, we weren't provided much to go on, so i'm treating *everything* as untrusted
16:17 MTecknology or is the minion setting tokens in storage untrusted?
16:18 MTecknology If you can't trust the things that are trusted to access your trusted token db, then I trust you won't find a trustable soultion
16:18 zer0def was about to bring that up
16:19 zer0def actually mean "treating as little as possible as trusted", but that doesn't change the conclusion
16:20 zer0def s/mean/meant/
16:20 MTecknology If you used the gpg option, you could store an encrypted version of the token, but that only keeps you from worrying about redis data being compromised
16:20 pcn Oh, where is the saltmaster's pubkey available?
16:20 msmith i don't trust any minion personally :)
16:20 aldevar left #salt
16:20 MTecknology I trust all minions enough to be my servers
16:20 pcn Do you have to get it from the filesystem, or is it made available via a grain?
16:21 MTecknology pcn: file system
16:21 pcn eh, not so bad if you're rolling this sort of thing together, I guess.
16:21 zer0def MTecknology: given how minion_1 is a fresh spun minion, that could've been achieved by pushing ciphertext through events, since it's basically meaningless unless you possess the private key
16:22 zer0def possess the private key and hijack the message, that is
16:22 MTecknology but it sounds like you don't even trust the master
16:22 zer0def i'm just trying to trust as little parties/machines as possible
16:23 zer0def which basically means mental gymnastics at this point
16:24 MTecknology right before it dives into total absurdity, it sounds pretty straight forward
16:24 zer0def oh, it already did so, twice.
16:29 pcn Can you ask the event cache for past events?
16:29 schemanic Hello. Can anyone help me with the best way to format and attach block devices to a salt minion at /var/log?
16:29 pcn I mean from a minion
16:39 zerocoolback joined #salt
16:40 zerocoolback joined #salt
16:44 zerocoolback joined #salt
16:45 rivyn Got salt-vim set up, thanks again
17:18 miruoy joined #salt
17:18 inad923 joined #salt
17:22 aruns__ joined #salt
17:27 nkuttler joined #salt
17:36 wongster80 joined #salt
17:51 ymasson joined #salt
17:54 rivyn I made a couple pretty simple SLS's that install & remove the pacemaker package (using Ubuntu 16.04).  The former does a pkg.latest and the last does a pkg.purged.
17:55 rivyn For some reason after I run the remove.sls, the install will no longer work because some lines are left in /var/lib/dpkg/statoverride from the previous instalation that cause dpkg to error
17:55 rivyn Is it advisable to use file.replace on /var/lib/dpkg/statoverride to remove the offending lines in the remove.sls?
17:56 rivyn seems like there should be a cleaner method
17:57 lordcirth_work rivyn, why do you need to remove pacemaker?
17:57 rivyn lordcirth_work: for anything we can install, we also want a method to remove/clean
17:58 lordcirth_work rivyn, for a specific reason, or on general principles?
17:58 rivyn if we are using pacemaker on some servers, and then stop and use something else to do the same thing or otherwise repurpose the servers, we don't want to leave it installed with a configuration that is no longer valid.
17:58 rivyn so both
17:59 rivyn it's "general principle" until it actually happens, but such things do actually happen.
17:59 lordcirth_work rivyn, why not run each service in a VM or container?
17:59 lordcirth_work Anyway, if you can clearly specify the lines in question, it doesn't seem that bad.
17:59 rivyn I'm currently rolling out pacemaker as a replacement for something else across dozens of servers
17:59 lordcirth_work container template + states = desired config.
17:59 rivyn in the future the same will likely happen again
18:00 rivyn the servers are VM's.  That doesn't make a difference?
18:00 rivyn shouldn't I be able to use salt to install and remove a package?
18:00 whytewolf rivyn: looks like ubuntu themselves pretty much say yeah go ahead and edit the file see https://askubuntu.com/questions/335538/unknown-user-in-statoverride-file?utm_medium=organic&amp;utm_source=google_rich_qa&amp;utm_campaign=google_rich_qa
18:01 rivyn alright, thanks whytewolf
18:01 lordcirth_work Of course you can.  The question is whether that's the right way.
18:01 rivyn I don't like it but it's easy enough to do :)
18:01 rivyn lordcirth_work: I really don't understand your criticism.  Using apt to install or remove a package on a debian/ubuntu system is pretty widely recognized as the "right way" to do such operations.
18:02 lordcirth_work Under the traditional model, yes.  Docker fans would say it is obsolete, though I don't entirely agree.
18:02 lordcirth_work And I wasn't intending to criticize, just checking for possible XY problems
18:02 rivyn Some folks like containerization and that's fine, but I don't have experience with that and we're not currently using any form of it here.
18:03 rivyn I can't really argue for or against it as I have very little knowledge about it
18:03 Miuku What, you aren't container religious yet? To the container re-education camp!
18:04 rivyn Miuku: I have higher-priority things to learn!
18:04 Miuku They'll whip the use-the-right-tool-for-the-job out of you and replace it with buzzwords!
18:04 rivyn too much stuff in the world at times
18:04 mpanetta joined #salt
18:04 om2 joined #salt
18:06 whytewolf container, lazy code word for adding a layer of comlexity because deveopers don't understand systems
18:06 whytewolf but that is just my feelings on the subject
18:07 lordcirth_work It is sometimes used for that purpose :P
18:07 lordcirth_work But having the application you run no longer being tied to a specific box in a specific building is quite convenient.
18:08 wryfi i blame languages with big unwieldy runtimes, for that problem.
18:08 MTecknology the containers that docker creates, imo, are suitable only for personal dev projects and automated testing.
18:08 Miuku Oh my, so many unbelievers. <blows docker re-education corps whistle>
18:08 lordcirth_work Especially since there are so many services these days that like to make changes to a system, but don't require nearly enough power to justify a 1U server all to themselves.
18:09 Miuku Seriously though, docking (not the dirty thing) can get really handy with stuff like GitLab.
18:09 wryfi any salt-cloud types around today? i'm working on a new salt-cloud driver, and trying to assess the cost-benefit ratio of doing it with libcloud.
18:09 MTecknology Miuku: in what way?
18:09 Miuku MTecknology: For testing purposes, as you said. We use GitLab -> Docker (test+deploy) -> Somewhere real where it breaks because my tests are bad.
18:09 Miuku :-)
18:10 wryfi the salt cloud docs suggest that libcloud integrations should be Super, Duper Easy! but when i look at the existing cloud drivers, only 3-4 of them use libcloud, and they are all big and unwieldy.
18:10 MTecknology wryfi: if you can implement a simple driver using libcloud that doesn't get big/clunky to handle lots of use cases, then you should definitely do that thing.
18:12 MTecknology Miuku: I too often see it being used for rolling dev/test/prod environments and it makes me want to slap them in the hollow think sphere with a clue bat.
18:12 wryfi MTecknology: i mostly agree in theory, but when i look in practice, i'm having a hard time justifying the extra effort. i have a rest api client to write against the API i'm integrating with. then a libcloud wrapper. then a salt-cloud wrapper around the libcloud wrapper. and the salt-cloud docs are very clear and concise about exactly how to implement a driver, whereas the libcloud docs consist of "look at some examples and figure it out," and the examples
18:12 wryfi don't even all have consistent methods on the objects they implement. it's an annoying developer experience.
18:13 MTecknology oh- is the question about not using libcloud when not needed?
18:13 wryfi yeah, i have a green field project, here.
18:14 MTecknology because if you can write something more stable/straight-forward with less work than it takes to implement the library features yourself... screw the overhead.
18:15 wryfi that is the direction i'm leaning, i just wanted to make sure i wasn't missing something obvious that would make the libcloud development easier or more compelling.
18:15 lordcirth_work Why does  "salt '*' test.ping -t 1 " take more than 1 second to return?
18:15 whytewolf this is the day of wtf projects. Docker, libcloud.
18:15 wryfi lol
18:15 Miuku MTecknology: We're mostly a media business (broadcasting, DVB/IP) so we can't use containers for most of the stuff. Or rather, it makes no sense to do so.
18:16 MTecknology I don't mind containers the way lxc does them.
18:16 lordcirth_work Miuku, ah, so your apps are actually bound to specific hardware for IO?
18:17 MTecknology hardware token to authorize the use of hardware?
18:17 Miuku lordcirth_work: Yeppers, which also eliminates using vmware and other virtualization platforms. Salt on the other hand is great for our use, makes it a breeze to roll out new systems and keep them in line.
18:17 MTecknology software*
18:23 jpsharp You can do USB passthrough for USB authentication tokens on VMware.  But it nails that VM to that host, so you can't vmotion it if you need to.
18:25 aldevar joined #salt
18:26 whytewolf might as well not us vmware at that point.
18:26 whytewolf [xenserver also has a usb passthrough mode]
18:27 coredumb is there a modern HV that has not ?
18:27 lordcirth_work Does --state-verbose=False not work for --out=json?  Do I have to process that out myself?
18:27 jpsharp i did it to aggregate machines.  There was no need to run eleventy thousand servers for small processes that needed USB keys.
18:29 jpsharp Only downside to it was that every time I had an application problem, the immediate reaction was "it's because of the virtualization".
18:31 Miuku You should see what VMware thinks of PCI-E broadcast cards in passthrough mode. I like purple, it's a nice colour - just not when it comes to ESX :-)
18:32 MTecknology I just ran into an interesting thing... I'm using {% for app in app_list %} {# X #} {% include include_path ~ '/' app ~ '.sls' ignore missing %} {% endfor %}   I want to conditionally include something at {# x #}, but only if the include succeeds.
18:34 MTecknology Is it possible to do something like assigning the included data to a variable that I could use an if for?  {% set foo = include(...) %} {% if foo %}?
18:36 esteban joined #salt
18:36 jpsharp Ah, never had the need to stick DVB cards in servers.  That's what we had racks of Scientific Atlanta receivers for.
18:43 omgitsmeagain joined #salt
18:44 omgitsmeagain left #salt
18:46 jas02 joined #salt
18:51 MTecknology an empty key would still exist in rendered pillar, wouldn't it?..
18:56 Neighbour empty as in `None`?
18:56 dendazen joined #salt
18:57 MTecknology that would depend on the answer to my question..
18:57 Neighbour well python does seem to accept None as a dict key
18:57 MTecknology ah, you're answering the second- sorry, my mind was on the first
18:58 MTecknology ya, I think that's what happens- it renderes in the stack w/ 'key: None'
18:58 schemanic anyone know why the salt-minion service would stop after initial highstate?
18:59 MTecknology because your states play with the minion process and break it?
18:59 schemanic I'm using this: https://github.com/saltstack-formulas/salt-formula/blob/master/salt/minion.sls
19:00 MTecknology welp- you know my opinion of formulas. I'm gonna step outta this one
19:01 schemanic hmm
19:03 Miuku You love them and think they're the best thing ever?
19:03 hillna joined #salt
19:03 schemanic I think they get me farther than inventing the wheel for what I'm trying to do
19:04 schemanic Right now I'm just trying to understand why the salt minion gets turned off and doesn't come back
19:07 esteban joined #salt
19:08 cewood joined #salt
19:18 sjorge joined #salt
19:21 schemanic I am completely stumped at this point. My salt-minion service is just dying after initial highstate and I don't understand why. Can anyone help me with this?
19:21 MTecknology Is there any example of accessing jinja functions from py in an sls file?
19:22 MTecknology or is there a jinja way to evaluate if a pillar sls file exists?
19:22 babilen schemanic: Run the highstate in debug mode on the minion with "salt-call -ldebug salt-minion" and also include output of "systemctl status salt-minion.service" along with "journalctl -u salt-minion.service" on a pastebin
19:23 babilen err
19:23 babilen "salt-call -ldebug state.apply" naturally
19:23 babilen MTecknology: You can call file.exists, but I mostly get away with "ignore_missing: True" these days
19:23 jas02 joined #salt
19:24 schemanic babilen where are you asking me to run the latter commands, all on the minion?
19:25 MTecknology file.exists won't work because this is in git, and ignore_mising: won't work because there's a big pile o' magic
19:25 babilen schemanic: aye
19:25 babilen MTecknology: What's wrong with magic?
19:26 schemanic thanks babilen. I'm spinning up some bare servers to work from
19:26 MTecknology the magic is creat, but it does make it not possible to use ignore_missing
19:26 babilen schemanic: Just tricky to debug without actual errors/data/output
19:26 MTecknology great*
19:26 schemanic babilen, I understand. I'm just having a devil of a time trying to observe whats going wrong
19:26 babilen Sure
19:27 babilen You might want to cut down your minion's highstate to just the tricky bit, so that we don't end up with a bazillion unrelated states
19:27 MTecknology does -ignore_missing: work in non-top files? I thought that was top only.
19:27 babilen Think so too
19:28 babilen Are you after an include + ignore_missing?
19:28 MTecknology nope
19:28 MTecknology I want to not include the line above if missing
19:30 babilen https://github.com/saltstack/salt/issues/16687
19:30 babilen Ah .. no idea and dinner time
19:30 MTecknology I tried to tweak this to use ignore_missing and it just breaks things hard
19:30 Hybrid joined #salt
19:31 schemanic babilen, I don't have journalctl on this host
19:31 babilen That's fine .. any log output would do
19:31 babilen But I'm off now .. might have a chance to look later, but others will surely appreciate the information
19:32 schemanic 'salt-call -ldebug salt-minion state.apply' doesn't work
19:33 schemanic it keeps saying 'salt-minion' is not available
19:33 esteban joined #salt
19:43 jas02 joined #salt
19:43 DammitJim joined #salt
19:45 inad924 joined #salt
19:47 jas02_ joined #salt
19:47 lordcirth_work schemanic, salt-minion isn't an execution module
19:48 lordcirth_work Just run salt-call -ldebug state.apply
19:52 MTecknology Woohoo! I think I got it! :D
19:53 sjorge joined #salt
19:55 babilen MTecknology: hmm?
19:55 babilen schemanic: sorry, I corrected it later, but you appear to have mixed the commands
19:55 MTecknology babilen: If a file is not included because it's missing, then the lines above what gets included will also not exist.
19:56 babilen But does it not explode in a burning ball of fire?
19:57 MTecknology it seems to work as perfectly as I could possibly expect it to. I'm even trying to break it.
19:57 jas02 joined #salt
19:58 MTecknology I always advocate keeping as much jinja out of your sls files as possible, and especially keep the logic out.
19:58 babilen +1
19:58 MTecknology I took that and murdered it to a thousand pieces
19:58 MTecknology https://gist.github.com/MTecknology/43b260ba6c23f26f6363e805474826ac
20:01 strobelight joined #salt
20:14 jas02 joined #salt
20:17 jas02 joined #salt
20:18 schemanic so to my frustration calling salt call --local ran properly and did not result in a stopped salt-minion
20:18 schemanic at this point I'm considering just writing a bash script to auto-start salt-minion if it finds it off
20:26 layer joined #salt
20:28 layer left #salt
20:28 layer joined #salt
20:29 jas02 joined #salt
20:29 lordcirth_work Anyone have some jq scripts for parsing salt returns they could share?
20:29 Neighbour MTecknology: that is some piece of work....nice :)
20:30 MTecknology thanks, it sure broke my brain coming up with it, but it seems simple now.
20:31 Neighbour it's a mutant jinja turtle though :P
20:31 Neighbour but perhaps that's why I like it so much
20:32 MTecknology I don't like it, but not sure how I could do it better
20:34 layer joined #salt
20:37 shpoont joined #salt
20:37 jas02 joined #salt
20:41 irated joined #salt
20:42 ponyofdeath joined #salt
20:43 jas02 joined #salt
21:10 aldevar left #salt
21:13 exarkun joined #salt
21:19 jas02 joined #salt
21:30 nixjdm joined #salt
21:38 shpoont joined #salt
21:38 karlthane joined #salt
22:07 mauli joined #salt
22:18 jas02 joined #salt
22:38 mrueg joined #salt
22:50 canci joined #salt
23:13 K0HAX ok, question.. I'm trying to deploy a cloud mapfile to VMware, and the VM it is deploying is behind NAT relative to the Salt Master. I created a ~/.ssh/config file with a ProxyCommand, and when I ssh to the IP address that the VM ends up on, it works, but when I run `salt-cloud -m myMap`, salt can't ssh to it and times out. Any ideas?
23:51 armin joined #salt
23:56 hemebond Does your master have the rquired SSH key/credentials?
23:59 digi_murali joined #salt

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary