My best practice with cron is to not use cron if possible. Systemd has timers with practically the same functionality and you'll never worry about:
- Where to put the logs, they just go into the system journal. Easy to check, and if you already have alerts/monitors/whatever for the journal any new job integrates trivially.
- Cron shell doing weird things.
- Easy switching on/off of each timer.
- Easy monitoring of when tasks are executed and the results.
- Jobs don't get started if there's one already running (I think)
> My best practice with cron is to not use cron if possible.
Absolutely.
I only randomly learnt about systemd timers thanks to a comment I read here on HN once. Since that point I've never looked back, its simply awesome.
The main reason is its enabled me to remove a whole load of boilerplate from my scripts because systemd takes care of it for me.
So my scripts are now shorter, more readable, and less prone to bugs.
> Jobs don't get started if there's one already running
Yes. This was the biggest selling point for me.
On cron, I had a ton of boilerplate using all sorts of tricks to avoid the race condition of the script being run again before it had finished its prior execution.
With systemd ? Remove the boilerplate, and, erm that's it. IIRC I don't think it even needs any special lines in your systemd unit file, it Just Works (tm).
> use flock command before your cronjob, hardly a ton of boilerplate
Let's not go there shall we.
Yes I know about flock, that's what I used.
But if you are writing code correctly, you still end up with boilerplate. For a start, you need to write extra code to check the error status returned from flock.
And then there's the overall point I was making. My "ton of boilerplate" referred to the overall boilerplate, i.e. flock was just one element, so then consider all the other boilerplate that was required to do all the other stuff that systemd does out of the box.
Frankly, its a no-brainer. Systemd wins every time.
Separating the schedule from the process might be an annoyance to newcomers... But I promise it's worth it
It's two files to template instead of one line, but it fixes essentially every lesson we all learned with cron. $PATH being incomplete, locking, etc
Another nice benefit is dependency handling.
If your 'cron job' requires networking you have to write boilerplate code or let it fire/fail endlessly. Pray the processes don't linger/stack or otherwise make a mess.
If a timer needs networking, set After= and Requires= to networking.target or whatever is appropriate for your distribution - and you're golden.
Even better if the job acts against another actual systemd service.
Using systemd-networkd you can tie the timers to specific interfaces. [but this handles NAT weirder than you're probably used to, if a host acts as a router prepare to adjust]
For non-system administration stuff the best practice goes even further, actual applications probably shouldn't use either cron or systemd timers as a method of scheduling internal jobs. At most an application (if not already a service) should have a single timer on a short interval, but the application itself should internally manage jobs and scheduling. The last thing you want is state being stored in the OS files.
One of old school Linux's nastiest antipatterns is applications that unnecessarily depend on global configuration. It was everywhere, and it makes automated setup a major hassle.
Lots of small self hosted things get written as if they were meant to be set up by IT professionals, and even then some of it is an unnecessary hassle even if it is meant for a commercial setting.
What's even worse is pre systemd stuff that would put multiple unrelated services in one file. Systemd does it right because all your services are separate files with very little global anything that isn't inherently global.
It's just like the LAMP stack. There are almost no PHP apps, just PHP code you can assemble into an app yourself with other daemons, usually with some subtle ways to get it wrong.
Wheras more recent stacks are often unzip and run or apt-get and run with only a small amount of local configuration.
If you happen to be on a Linux system with a properly configured mta (mail daemon), which you are if cron is able to send emails, you can just pipe the output of your command to sendmail
It would have been nice until extra logic comes in: cron doesn't send an email when there is no output.
Which is exactly what I want! Error? I must know. Output? If I left any, then it's a notification, and I must know. No output? A maintenance task succeeded, nothing to report.
> The one reason I don't use systemd timers is that they don't have an automatic, reliable way to send me an email with their output.
I can't remember the exact unit file syntax but you can define Before/After hooks, so you can point that to a script that emails people (or in my case pushes to a notification queue).
Dang, true. Didn't catch that.
I think sometime I had set cron to run a script every day at a certain time and bypassed every other day with a simple .txt file.
Top of my script was a simple if-clause, when alternate-day.txt is found, it will be removed and the script runs. If no .txt is found it will be created and the script exits. A bit lame, but it worked and was quick :)
Interestingly, and on further consideration, using date with %w (as in the example I provided) suffers from the same problem, lol! Creating a file as you did should work nicely. Or a fair bit more arithmetic could be done using date's %s, not my cup of tea though.
As mentioned, the next problem appears when the job takes longer than two days, which requires using flock. I often prefer to use mkdir instead of flock, provided your mkdir is atomic and POSIX compliant... which is usually the case AFAIK. E.G. mkdir a-lock-dir || exit;... rmdir a-lock-dir
Ultimately I don't have too many complaints about cron. But, like anything, it has it's own quirks. Indeed a handy article.
yeah, calendar manipulation is trickier than it looks. If you use day-of-week, you have an odd number of days. If you use day of month, many months have an odd number. If you use day of year (%j), most years have an odd number.
There's many such examples - every 1, 2, 3, 4, 6 hours works - but every 5 hours doesn't. But every 2 days and every 14 days are requirements I've met IRL.
My problem is that I always forget the complicated dance of creating a one-shot service file, then a timer of the same name, then enable then start. And is it --user? Where do they go again?
I should write a little bash script that takes a name, a command, and a schedule and just sets it all up. Has anyone already done something like that?
They forgot the most important one IMO: document the intended duration and schedule of your cron in a comment above. Most people can't decipher the cron syntax by heart or without making a mistake, and among those most people, there is probably you when you are tired.
Of course, there is always crontab guru (E.G: https://crontab.guru/#*/12_*_*_\*), but still, it's easier to find out with a comment next to it. Besides, it documents the intent, which may shows that there is a mismatch with the reality.
I fully expected one item to show up here, since this was shared at such an auspicious moment: don’t schedule any daily tasks between 1 and 2am local time, to avoid DST changes from either skipping or running them twice.
(Unless cron has gotten smarter since last I used it, which was admittedly at least a decade ago.)
Vixie cron (in the RedHat realm, written by Paul Vixie) addresses this.
Local time changes of less than three hours, such as those caused by
the Daylight Saving Time changes, are handled in a special way. This
only applies to jobs that run at a specific time and jobs that run with
a granularity greater than one hour. Jobs that run more frequently are
scheduled normally.
If time was adjusted one hour forward, those jobs that would have run
in the interval that has been skipped will be run immediately. Con‐
versely, if time was adjusted backward, running the same job twice is
avoided.
Time changes of more than 3 hours are considered to be corrections to
the clock or the timezone, and the new time is used immediately.
It is possible to use different time zones for crontables. See
crontab(5) for more information.
Vixie crontabs also allow @reboot that enables any user with cron access to run programs at boot, which can be quite useful.
There other variables in cron besides MAILTO, such as MAILFROM and SHELL. Checkout "man 5 crontab" for a concise listing.
If cron runs on more than one host, it can be useful to add some randomness to the time to avoid the same job being run on exactly the same time everywhere. Red Hat-family distributions have RANDOM_DELAY variable for this. Or just add the delay when Ansible or a similar tool installs the job from version control. (Which you use, right?)
In addition, I also want to see the status of the transient cron job and see how much time is left before it executes, like `systemctl list-timers temporary.timer` would show.
At some point when the number of boxes and number of cronjobs become too high, I suggest looking at Rundeck as a central scheduler (not affiliated with PagerDuty, I actually work for a competiting company) or something similar.
Also, you want to make sure your job has been running. I've used a selfhosted instance of healthchecks.io in order to warn me if something has not run.
To say more: I have sometimes had to resort to writing a wrapper script just for the purpose of gatekeeping what stdout/stderr a task in cron does (and ensure the exit code is properly (non-)zero.
At one former job where I had a lot of cron jobs, I even wrote a re-usable wrapper utility for it.
Many unix command line executables are pretty bad at it (especially if you deal with anything proprietary), so I get why people resort to just throwing away output from cronjobs, although it is a problematic lack of observability, true.
It's because people want to test whether their command is actually successfully executed by cron because basic functionality like starting a job immediately isn't available and then they forget to revert that change.
You have to use systemd timers for that... It's completely overkill (2 files each with a dozen lines) but it works much better in practice.
I don't think that's the problem, I just think its really easy to forget to set it to 0. I literally lol'd when I just happened to be reading the docs here just now: https://rancher.com/docs/k3s/latest/en/installation/install-... and saw it under the example cron for "--etcd-snapshot-schedule-cron."
Locking is non-trivial, don't just use flock. Use the maildir method and check for stale locks; it's atomic, portable, and NFS-safe.
Using Ohmycron is better than nothing, but it doesn't handle timeouts and stale locks and some other useful things. Seems like we still need a better universal wrapper for cron jobs.
- Where to put the logs, they just go into the system journal. Easy to check, and if you already have alerts/monitors/whatever for the journal any new job integrates trivially.
- Cron shell doing weird things.
- Easy switching on/off of each timer.
- Easy monitoring of when tasks are executed and the results.
- Jobs don't get started if there's one already running (I think)
- Better error messages when you mess the syntax.
- More possibilities for scheduling.
- Launching the job now if you need it to.