Perl 6 - the future is here, just unevenly distributed

IRC log for #salt, 2018-03-27

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to UTC.

Time Nick Message
00:12 armin_ joined #salt
00:33 _maniac_ joined #salt
00:34 _maniac_ Hi. Does anyone know recipe to run raw ssh command from salt runner?
00:39 zerocoolback joined #salt
00:49 _maniac_ ok, nvm: cmd.run works for me. although slower.
00:58 onlyanegg joined #salt
01:06 justanotheruser joined #salt
01:07 justanotheruser joined #salt
01:21 eseyman joined #salt
01:55 ilbot3 joined #salt
01:55 Topic for #salt is now Welcome to #salt! <+> Latest Versions: 2016.11.9, 2017.7.4 <+> RC for 2018.3.0 is out, please test it! <+> Support: https://www.saltstack.com/support/ <+> Logs: http://irclog.perlgeek.de/salt/ <+> Paste: https://gist.github.com/ <+> See also: #salt-devel, #salt-offtopic, and https://saltstackcommunity.herokuapp.com (for slack) <+> We are volunteers and may not have immediate answers
01:56 katoline joined #salt
01:57 katoline left #salt
02:22 shiranaihito joined #salt
02:24 zerocoolback joined #salt
02:52 zerocoolback joined #salt
03:18 ponyofdeath joined #salt
03:42 inetpro joined #salt
03:45 eekrano joined #salt
03:52 evle joined #salt
03:57 dograt joined #salt
04:04 inetpro joined #salt
04:06 tiwula joined #salt
04:07 moop_ joined #salt
04:18 justan0theruser joined #salt
04:19 justan0theruser joined #salt
04:27 hemebond joined #salt
04:32 indistylo joined #salt
04:32 zerocoolback joined #salt
04:56 Hybrid joined #salt
05:22 Hybrid joined #salt
05:34 sauvin_ joined #salt
05:38 aruns joined #salt
05:38 mavhq joined #salt
05:54 Guest73 joined #salt
05:55 aruns__ joined #salt
05:58 tyx joined #salt
06:16 marwel joined #salt
06:31 shpoont_ joined #salt
06:34 viber- joined #salt
06:35 sauvin_ joined #salt
06:55 inetpro joined #salt
07:00 aldevar joined #salt
07:02 rahav joined #salt
07:03 rahav folks, i have 2.5k minions connected to master and im on tcp transport im seeing that TCPPubServerChannel consumes high memory about 2.6GB of RAM. Is this expected
07:03 aruns__ joined #salt
07:08 jas02 joined #salt
07:24 Guest73 joined #salt
07:30 marwel joined #salt
07:31 pjs_ joined #salt
07:31 Tucky joined #salt
07:33 scooby2_ joined #salt
07:37 jrenner joined #salt
07:41 darioleidi joined #salt
07:49 Guest73_ joined #salt
07:52 Danny159 joined #salt
07:53 Nazca joined #salt
07:53 darioleidi joined #salt
07:54 hemebond joined #salt
07:58 inad922 joined #salt
07:59 noby joined #salt
08:04 stan joined #salt
08:06 Ricardo1000 joined #salt
08:07 Danny159 joined #salt
08:09 sybix joined #salt
08:23 inad922 joined #salt
08:33 Mattch joined #salt
08:37 zomane joined #salt
08:39 oida joined #salt
08:42 inad922 joined #salt
08:53 Tyrant joined #salt
08:56 jrenner left #salt
08:59 jrenner joined #salt
09:04 pf_moore joined #salt
09:05 inad922 joined #salt
09:09 jxs1_ joined #salt
09:19 k1412 joined #salt
09:24 svg_ joined #salt
09:41 cyteen joined #salt
09:48 hemebond rahav:  2.5k minions on a host? Nice. What are the specs of the master?
09:49 hemebond (I have no idea about the memory issue)
09:53 zerocoolback joined #salt
09:55 OliverUK left #salt
10:17 CrummyGummy joined #salt
10:19 Hybrid joined #salt
10:23 rahav hemebond : master is running on a vm with 16gb ram and 8vcpu
10:32 indistylo joined #salt
10:41 cyteen joined #salt
10:46 aruns joined #salt
10:56 mavhq joined #salt
11:05 inad922 joined #salt
11:06 shpoont_ joined #salt
11:07 jrj joined #salt
11:12 Guest73 joined #salt
11:18 evle joined #salt
11:18 pcn hemebond: I also don't know about that. What are the effects of that memory use? Are you troubleshooting some issues?
11:22 Miuku Still takes less memory than my Chrome..
11:44 exarkun that's because nothing can take more memory than chrome
11:45 exarkun chrome is defined as that entity in the universe which consumes the majority of memory
12:10 mahafyi joined #salt
12:11 Nahual joined #salt
12:17 shpoont_ joined #salt
12:50 jxs1_ joined #salt
13:01 inad922 joined #salt
13:12 colin_stubbs joined #salt
13:13 colin_stubbs joined #salt
13:15 edrocks joined #salt
13:16 exarkun How do folks deal with authentication with gitfs?
13:17 exarkun Surely I don't want to check a private key or password in to my git repo inside my salt configuration
13:20 babilen gpg encrypted pillars or vault sdb/pillar are approaches
13:21 exarkun how/when do you do the decryption?
13:45 darix joined #salt
13:45 cyteen joined #salt
13:48 mchlumsky joined #salt
13:50 GrisKo joined #salt
13:53 KyleG joined #salt
13:53 KyleG joined #salt
13:56 racooper joined #salt
14:00 thelocehiliosan joined #salt
14:07 wedgie joined #salt
14:17 tyx joined #salt
14:22 scooby2_ left #salt
14:22 scooby2 joined #salt
14:27 darix joined #salt
14:27 babilen exarkun: The master is decrypting the pillar whenever it is rendered (gpg pillar). See https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html for details
14:31 fnords joined #salt
14:41 mchlumsky joined #salt
14:43 tiwula joined #salt
14:46 rivyn good morning fellas
14:47 babilen afternoon :)
14:49 rivyn that too :)
14:56 DanyC joined #salt
15:09 q1x joined #salt
15:10 bluenemo joined #salt
15:17 xet7 joined #salt
15:22 dezertol joined #salt
15:25 jas02 joined #salt
15:27 jas02 joined #salt
15:27 evle joined #salt
15:38 immune2iocane joined #salt
15:39 inad922 joined #salt
15:42 jas02 joined #salt
15:44 jas02 joined #salt
15:48 dezertol joined #salt
16:00 cgiroua joined #salt
16:11 rivyn zer0def: I spoke with the maintainer of pg_ctlcluster & friends, and he's as confused as I am as to why the hang is happening from Salt but not manual execution.  Is there any further insight you could offer?
16:12 rivyn Also, wanted to ask more about why you preferred file.managed to file.recurse
16:12 zer0def `file.recurse` does just a recursive copy, with `file.managed` i can provide templates that are rendered on-minion
16:13 zer0def oh, actually, that's not true, now that i've reviewed the function arguments
16:14 zer0def i just prefer it over recurse because more often than not i only modify a handful of files
16:18 zer0def might be because i prefer explicit statements on what i'm changing, as opposed to "hey, i'm just recursing this dir here"
16:20 zer0def speaking of pg_ctlcluster, apparently that's a debian thing, so in case you'd like to support other distros, you might want to consider switching to pg_ctl
16:20 buumi joined #salt
16:21 DanyC joined #salt
16:22 DanyC_ joined #salt
16:35 inad922 joined #salt
16:50 edrocks joined #salt
16:55 jhauser joined #salt
17:00 rivyn zer0def: yeah, debian & ubuntu.  But all we are using here is Ubuntu.
17:01 rivyn zer0def: I just realized that you can use names like postgresql@10-main to manage a specific cluster with the service command - so I'm wondering if I shouldn't just replace the cmd.run pg_ctlcluster bit with a service.started postgresql@10-main
17:01 rivyn I think you mentioned that yesterday as well but I didn't grasp it fully at the time
17:02 zer0def well, i wasn't sure that'd actually work
17:03 zer0def it also relies on your service manager being systemd, so there may be some discrepancies until 14.04 gets EOL'd
17:05 whytewolf well he did check if it was ubuntu 16.04 in his jinja so that shouldn't be a problem
17:06 rivyn zer0def: the systemd requirement is okay - our PostgreSQL servers are exclusively Ubuntu 16.04 currently.
17:06 zer0def eh, i'd love to have distros so consistent ;)
17:07 rivyn zer0def: the only reason I need to start the cluster is so that I can create db users and make a basebackup.  After that it can be stopped.  In my older SLS scripts, I had the service start, then stop again at the very end.  However, that meant if I ran the SLS again, it would start/stop the cluster every time which was undesirable.
17:07 dezertol joined #salt
17:08 rivyn ultimately, I should make the SLS only touch the cluster when it's actually doing an initialization the first time, then stop it at the end and leave management of the service to pacemaker
17:08 zer0def that sounds like `service.running` needs a `prereq` on `postgres_user`, btw
17:09 zer0def then have `service.dead` having an `onchanges` requisite on `postgres_user` states
17:09 rivyn I started with one huge SLS which called a bunch of complex BASH scripts that wasn't safe to run against a server it had already been run against.  That's 14 SLS's now without any shell scripts involved.  So as much of improvements as I still have to make, it's an improving work-in-progress...
17:10 zer0def the tl;dr is primarily: if you can use `service` states, they're probably spare you a lot of headaches, since someone already went through implementing those
17:10 zer0def s/y're/y'll/
17:11 rivyn yeah, that's what I figured and am hoping - about to give it a shot
17:11 Edgan zer0def: I always create users before pkgs to avoid the service already being started. I also create my own users to get more consistently than the packages will provide. Only ever run into one packages that wasn't written to deal with an existing user.
17:11 zer0def Edgan: are we talking system users or postgres db users?
17:12 Edgan zer0def: system users. For postgres users, you have to edit the file and restart postgres?
17:12 zer0def yeah, system users are largely irrelevant to the topic with rivyn
17:13 rivyn Edgan: I am doing the same - Ubuntu otherwise picks a different UID depending on what else got installed first, which doesn't play nice with NFS
17:13 zer0def ideally you have pg_hba already handling which users can login, but to add pg users, you need to have a running cluster, and apparently rivyn's requirement is to have the cluster stopped as soon as it's created and users are provisioned
17:14 rivyn well I don't *have* to stop it, but it seems cleaner to
17:14 rivyn since we are layering on pacemaker/corosync for cluster management after PG is set up.
17:14 Edgan rivyn: yeah, and sometimes you will end up with 105:401 include of uid:gid
17:15 rivyn Edgan: yeah, we have it hardcoded to 800:800
17:15 Edgan rivyn: I would be interested in seeing your formula afterwards if you can open source it. We were just talking about pg and corosync yesterday.
17:15 zer0def if you intend to user it as some sort of template, since a downed cluster will probably take a while until it syncs (unless you're sharing storage)
17:15 Edgan rivyn: I keep a file of first party(our apps), and third party(everyone else) uids.
17:16 rivyn Edgan: well you might be unimpressed with the SLS currently - this is my first time using salt, so it's a work-in-progress.... :P
17:16 zer0def Edgan: for PG HA?
17:16 Edgan rivyn: I am more interested in getting corosync working. I have fought with it before. The best I got working was keepalived.
17:17 Edgan zer0def: yes
17:17 rivyn Pacemaker/corosync are easy once you get the pacemaker configuration right
17:17 Edgan rivyn: This was a few years ago in AWS. It was just not working for me.
17:17 zer0def i think i have something cobbled together using repmgr triggering pgbouncer updates which serve as a front to accessing all pg instances
17:18 rivyn Edgan: check out https://github.com/ClusterLabs/PAF if you haven't already
17:18 Edgan zer0def: Reminds me of tungsten for mysql
17:18 zer0def repmgr obviously handling tracking of whether the master is up and which slave is most up-to-date to failover to
17:18 Edgan zer0def: Cool, thanks.
17:18 schemanic joined #salt
17:18 rivyn I do want to play with repmgr, keepalived, haproxy, and maybe some others as possible alternative setups in the future
17:19 rivyn but that's a ways out
17:19 rivyn you can use pacemaker/corosync to manage a mysql cluster as well.  That's our plan once we get the PG stuff solid
17:19 ckonstanski left #salt
17:19 zer0def as far as i can tell, repmgr already handles what keepalived would, assuming haproxy would've been your gateway
17:20 zer0def i prefer pgbouncer primarily because i could relay read queries to slaves and have nearly exclusively writes hit the master
17:22 zer0def tried pgpool-ii, too, but it doesn't like anything else poking in it's business
17:24 rivyn we are using pgpool presently, and have loads of problems with it.
17:24 zer0def in which case i encourage repmgr and placing pgbouncers in front, which would be managed by the former
17:24 rivyn that's largely due to past employees who wrote a bunch of crappy scripts that it attempts to use for failover and such that just don't work.  But we've also measured significantly better performance by NOT going through pgpool
17:25 rivyn with pgbouncer, I think you have to use a separate connection from the app for read queries in order to load balance them between the standbys?
17:25 rivyn the "advantage" of pgpool is that it parses each query and figures out whether it's a read-only query or not.  I quote that because it's probably a source of problems.
17:26 zer0def well, i might be doing it badly, but i define two aliases for every database - one for writes that has only the master as it's backend, one for reads having all slaves
17:26 rivyn we have a bunch of apps - some could use multiple connections to split up reads and writes, but many cannot.
17:26 zer0def in the case of those that cannot, i guess i'd just shove them through the writing alias
17:27 rivyn that said, a single PG server outperforms pgpool distributing reads to 5 PG servers.
17:33 zer0def in either case, i highly recommend repmgr+pgbouncer for HA, unless you're willing to play with BDR, XL or logical replication
17:34 zer0def with citus for sharding, if necessary
17:36 rivyn *nods* I do like pgbouncer, that'll be something we add SLS for after the base PG+Pacemaker/corosync stuff is fully baked
17:39 rollniak joined #salt
17:40 DanyC joined #salt
17:46 rivyn zer0def: using the @<version>-<cluster> syntax with salt's service module appears to work great on the initial run.  I added the stop to the end as well and it does everything as expected
17:47 rivyn However on subsequent runs it tries to check if the db users are present and fails because it's not running, so I need a requisite there
17:48 zer0def oh yeah, i guess that's going to be a chicken-and-egg problem
17:48 rivyn I'll paste my current SLS
17:48 rivyn zer0def: https://ghostbin.com/paste/5zo52
17:49 zer0def in that case, i'd just assume that if the cluster already exists (`postgres_cluster` doesn't report changes), then there's no need to start the service, create users and stop it
17:49 rivyn I could add an onlyif: <check that cluster is running> to database_setup?
17:49 rivyn zer0def: correct
17:49 zer0def that would basically mean chaining `onchanges` starting on `postgres_cluster`
17:50 rivyn ideally the SLS would skip everything if it sees the cluster already there.  I'm not sure how to codify that
17:50 rivyn starting in postgres_cluster?  That's one of the first things
17:51 rivyn what would postgres_cluster have for it's onchanges?
17:52 rivyn I'm a bit confused about require vs. onchanges.  I have both currently
17:52 zer0def `cluster_running:service` has `onchanges` on `cluster_initialized:postgres_cluster`, `database_setup:postgres_user` the same on `cluster_running:service`, `cluster_stopped` would `require` on `database_setup`
17:53 rivyn cluster_running.service needs 2 onchanges you're saying?
17:54 zer0def the first statement basically starts the cluster's service only when it picks up on cluster changes, the second makes sure that, since the service is running, users exist; the final makes sure that the service is *always* stopped after users are created (if there was a need)
17:54 rivyn I'm not reading that message properly I don't think
17:55 rivyn `cluster_running:service` has `onchanges` on `cluster_initialized:postgres_cluster`  <-- already done, right?  So it needs an addition of the second onchanges?
17:55 zer0def https://ghostbin.com/paste/5dbbv
17:56 rivyn also is it supposed to be typed in as cluster_initialized:postgres_cluster?  Because I have postgres_cluster: cluster_initialized per yesterday's discussion.
17:56 rivyn thanks!
17:56 zer0def basicalled added `onchanges` and `require_in` in `database_setup` states
17:56 rivyn what's line 59?
17:57 zer0def not even sure how that got there, but you can disregard it
17:57 rivyn ok, I guessed as much :)
17:58 zer0def the premise here is that `cluster_stopped` will always run, but it *should* wait for `database_setup` to finish, if it needs to run
17:58 rivyn This looks to work well
17:59 zer0def to check whether it actually works well, try commenting out `unless` in `database_setup:cmd`
17:59 zer0def if it runs/reports changes, that's bad.
17:59 rivyn ah, so the unless should no longer be needed?
18:00 zer0def if it run/reports changes while `cluster_initialized:postgres_cluster` doesn't, that is
18:00 rivyn ahhhh crap.
18:00 zer0def i'd keep it there, i can imagine cases where keeping it would probably be beneficial
18:00 rivyn it worked fine for the first cluster, but attempting to add a second cluster to the server resulted in it hanging again
18:01 rivyn Last line:  2018-03-27 17:59:32,144 [salt.loaded.int.module.cmdmod:388 ][INFO    ][30662] Executing command ['systemd-run', '--scope', 'systemctl', 'start', 'postgresql@10-test.service'] in directory '/root'
18:01 zer0def let me guess, you checked for `main`, but another cluster tripped it
18:01 zer0def not sure what's going on there
18:02 rivyn hmm?
18:02 rivyn I added a second cluster named 'test' instead of 'main'
18:02 rivyn the SLS is doing everything right with regard to using the desired cluster name
18:03 zer0def well, i think requisites wouldn't change, if you swapped out those `service` states for previous `cmd` ones; it's inconvenient, that's for sure
18:03 zer0def the only case i'm not sure about is `cmd.run` vs `cmd.wait` on `cluster_running` with `onchanges`
18:04 rivyn zer0def: here's the pstree:  https://ghostbin.com/paste/56v8y
18:04 rivyn it's not as clear whether it's waiting for systemctl now
18:04 whytewolf zer0def: cmd.wait only works with watch. it doesn't nothing if there is no watch
18:04 zer0def thanks for the clear up, whytewolf
18:05 zer0def rivyn: i'd start looking in whatever syslog you have as to what's going on
18:05 zer0def clearly systemd's getting in the way in some manner
18:06 rivyn aha:
18:06 rivyn Mar 27 17:59:34 d-gp2-dbpg2-1 systemd[1]: postgresql@10-test.service: PID file /var/run/postgresql/10-test.pid not readable (yet?) after start: No such file or directory
18:07 zer0def ah, yes: `PIDFile=/var/run/postgresql/%i.pid` in the service file
18:07 zer0def aka, if you're overriding pidfile's location, whoever wrote the service file would like to interject and disagree with you
18:09 rivyn pidfile location is being set in the pg config, but I'm not sure why I did that.
18:09 rivyn what is the default location?
18:09 rivyn where are you seeing that PIDFile= bit?
18:09 zer0def `/lib/systemd/system/postgresql@.service`
18:10 rivyn thanks, I'm going to have to dive into systemd at some point soon I guess
18:10 schemanic I am having a bit of a problem. I need to provision new environments with salt cloud, but I don't want to upgrade my saltstack version yet. I am afraid that going from 2017.7.2 to 2017.7.4 will break some formulas of mine
18:10 aruns joined #salt
18:10 schemanic on amazon linux, it seems to be impossible to commission a server with a specific version of salt minion
18:10 zer0def schemanic: that's entirely dependent on your deploy script
18:11 schemanic zer0def, please elaborate
18:11 rivyn ahh, I found the problem.  the pidfile is hardcoded to 10-main instead of using jinja variables in the config
18:11 rivyn Amazon has their own linux distribution now
18:11 rivyn ?
18:11 schemanic the standard salt-bootstrap script attempts to reach a url that does not exist when attempting to pass it a specific version of saltstack
18:11 whytewolf amazon has had a linux distro for many years now
18:12 schemanic salt-cloud -p QAAP-C OTQAAPVM0000 --script-args "stable 2017.7.2" is what I'm running
18:12 zer0def ls
18:12 zer0def whoops, sorry
18:14 zer0def if i'm not mistaken, salt-cloud's default setting for `deploy_scripts_search_path` is `/etc/salt/cloud.deploy.d`, that's the location where you can provide your own deployment scripts; pre-bundled deployment scripts are located at `/usr/lib/python2.7/dist-packages/salt/cloud/deploy`
18:15 whytewolf schemanic: when was the last time you ran 'salt-cloud -u'
18:15 zer0def the script's filename without `.sh` extension is what you can provide to the `script` argument, be it in a cloud map or state
18:18 Bico_Fino Hello. How do I target a minion in a state running a module? Like token: {{ salt['module.something']('tgt: myminion') }} ?
18:19 deathscythe272 joined #salt
18:19 schemanic I ran it today
18:19 schemanic whytewolf,
18:20 whytewolf Bico_Fino: with out context. i think you might be looking for this https://docs.saltstack.com/en/latest/topics/mine/ because what you have asked does not make sense.
18:22 Bico_Fino whytewolf Basically I have a state that install an agent on a minion, but I need to grab the token from another minion. Going take a look at mine, thanks.
18:22 zer0def yes, by your description, it's definitely something you'd use mine for
18:23 Bico_Fino thanks!
18:24 aruns joined #salt
18:29 mahafyi_ joined #salt
18:33 mahafyi joined #salt
18:44 rivyn zer0def: sorry got a bit distracted.  I tried removing the unless: within database_setup:cmd.run, but it fails as it tries to run the cmd.run and fails because the directory already exists but isn't empty
18:55 eekrano joined #salt
19:03 davisj joined #salt
19:04 aldevar joined #salt
19:10 schemanic my minion wont talk to my master post upgrade
19:11 zer0def rivyn: on every or initial run?
19:12 schemanic joined #salt
19:12 schemanic /var/log/salt/minion mentions a salt.cryp error where the master has cached a key
19:12 schemanic salt.crypt* rather
19:13 rollniak joined #salt
19:14 rivyn zer0def: subsequent runs - initial is fine
19:14 rivyn when I call the SLS with the same args a second time
19:14 rivyn s/args/pillar/
19:21 immune2iocane joined #salt
19:24 DammitJim joined #salt
19:28 schemanic Does anyone know why /etc/salt/minion_id would be overwritten during a saltstack upgrade?
19:34 schemanic So, it seems that the minion_id file got overwritten when I upgraded saltstack on it
19:35 schemanic I don't know quite why that happened
19:35 edrocks joined #salt
19:41 sjorge joined #salt
19:59 DammitJim1 joined #salt
20:19 schemanic do states just stop trying to apply if the state times out? I'm giving my minions 5 minutes to run highstate initially and they ALWAYS timeout
20:22 rivyn schemanic: I haven't observed that, personally.
20:22 rivyn I've had states hang indefinitely until I kill the PID
20:23 rivyn perhaps there is a timeout that is just longer than I would have noticed
20:26 schemanic is there any way to diagnose why the state isn't running?
20:26 jas02 joined #salt
20:33 aldevar left #salt
20:36 jas02 joined #salt
20:43 tyx joined #salt
20:46 irated joined #salt
21:17 mechleg joined #salt
21:27 pjs_ left #salt
21:28 pjs joined #salt
21:31 tac_ joined #salt
21:31 ymasson joined #salt
21:57 thelocehiliosan joined #salt
22:56 mircea joined #salt
22:59 cholcombe joined #salt
23:01 Deadhand joined #salt
23:14 Guest73 joined #salt
23:24 immune2iocane joined #salt
23:26 pcn schemanic: you can add debug messages in jinja now IIRC
23:28 pcn https://docs.saltstack.com/en/latest/topics/jinja/index.html#logs
23:36 colin_stubbs joined #salt
23:38 mavhq joined #salt
23:56 infinity_ joined #salt
23:57 infinity_ I installed the macos py3 package just now on two macs and it basicaly made the computers unresponsive after typing salt-config. is there something i should know?

| Channels | #salt index | Today | | Search | Google Search | Plain-Text | summary