Crontab error log

As others have pointed out, cron will email you the output of any program it runs (if there is any output). So, if you don’t get any output, there are basically three possibilities:

1. crond could not even start a shell for running the program or sending email
2. crond had troubles mailing the output, or the mail was lost.
3. the program did not produce any output (including error messages)

Case 1. is very unlikely, but something should have been written in the cron logs. Cron has an own reserved syslog facility, so you should have a look into /etc/syslog.conf (or the equivalent file in your distro) to see where messages of facility cron are sent. Popular destinations include /var/log/cron, /var/log/messages and /var/log/syslog.

In case 2., you should inspect the mailer daemon logs: messages from the Cron daemon usually appear as from root@yourhost. You can use a MAILTO=... line in the crontab file to have cron send email to a specific address, which should make it easier to grep the mailer daemon logs. For instance:

MAILTO=my.offsite.email@example.org
00 15 * * *  echo "Just testing if crond sends email"

In case 3., you can test if the program was actually run by appending another command whose effect you can easily check: for instance,

00 15 * * * /a/command; touch /tmp/a_command_has_run

so you can check if crond has actually run something by looking at
the mtime of /tmp/a_command_has_run.

cron already sends the standard output and standard error of every job it runs by mail to the owner of the cron job.

You can use MAILTO=recipient in the crontab file to have the emails sent to a different account.

For this to work, you need to have mail working properly. Delivering to a local mailbox is usually not a problem (in fact, chances are ls -l "$MAIL" will reveal that you have already been receiving some) but getting it off the box and out onto the internet requires the MTA (Postfix, Sendmail, what have you) to be properly configured to connect to the world.

If there is no output, no email will be generated.

A common arrangement is to redirect output to a file, in which case of course the cron daemon won’t see the job return any output. A variant is to redirect standard output to a file (or write the script so it never prints anything — perhaps it stores results in a database instead, or performs maintenance tasks which simply don’t output anything?) and only receive an email if there is an error message.

To redirect both output streams, the syntax is

42 17 * * * script >>stdout.log 2>>stderr.log

Notice how we append (double >>) instead of overwrite, so that any previous job’s output is not replaced by the next one’s.

As suggested in many answers here, you can have both output streams be sent to a single file; replace the second redirection with 2>&1 to say «standard error should go wherever standard output is going». (But I don’t particularly endorse this practice. It mainly makes sense if you don’t really expect anything on standard output, but may have overlooked something, perhaps coming from an external tool which is called from your script.)

cron jobs run in your home directory, so any relative file names should be relative to that. If you want to write outside of your home directory, you obviously need to separately make sure you have write access to that destination file.

A common antipattern is to redirect everything to /dev/null (and then ask Stack Overflow to help you figure out what went wrong when something is not working; but we can’t see the lost output, either!)

From within your script, make sure to keep regular output (actual results, ideally in machine-readable form) and diagnostics (usually formatted for a human reader) separate. In a shell script,

echo "$results"  # regular results go to stdout
echo "$0: something went wrong" >&2

Some platforms (and e.g. GNU Awk) allow you to use the file name /dev/stderr for error messages, but this is not properly portable; in Perl, warn and die print to standard error; in Python, write to sys.stderr, or use logging; in Ruby, try $stderr.puts. Notice also how error messages should include the name of the script which produced the diagnostic message.

Var log cron error

Ubuntu Linux проверка ошибок cron

Где хранится информация о запуске заданий по расписанию? Для проверки используем Ubuntu Linux 12.04/14.04 LTS сервер. Как проверит, запущен ли cron и правильно ли он работает?

Как проверить запущен ли cron (планировщик заданий)? Используйте команду pgrep или ps

p grep cron ps aux | grep cron sudo service cron status sudo status cron

p s aux | grep cron

s udo service cron status

s udo status cron

Так же вы можете проверить файл /var/log/syslog используя команду grep что бы получить информации о работе планировщика cron

$ sudo grep —color -i cron /var/log/syslog

$ sudo grep —color -i cron /var/log/syslog

J an 17 17:43:21 planetvenus cron[229]: (CRON) INFO (pidfile fd = 3) Jan 17 17:43:21 planetvenus cron[240]: (CRON) STARTUP (fork ok) Jan 17 17:43:21 planetvenus cron[240]: (CRON) INFO (Running @reboot jobs) Jan 17 18:01:01 planetvenus cron[240]: (*system*cache) NOT A REGULAR FILE (/etc/cron.d/cache) Jan 17 18:01:01 planetvenus cron[240]: (*system*output) NOT A REGULAR FILE (/etc/cron.d/output)

J an 17 17:43:21 planetvenus cron[229]: (CRON) INFO (pidfile fd = 3)

J an 17 17:43:21 planetvenus cron[240]: (CRON) STARTUP (fork ok)

J an 17 17:43:21 planetvenus cron[240]: (CRON) INFO (Running @reboot jobs)

J an 17 18:01:01 planetvenus cron[240]: (*system*cache) NOT A REGULAR FILE (/etc/cron.d/cache)

J an 17 18:01:01 planetvenus cron[240]: (*system*output) NOT A REGULAR FILE (/etc/cron.d/output)

Где хранятся журналы работы cron на Ubuntu Linux?

Журналы хранятся в файле /var/log/cron.log. Вы можете настроить его следующим образом. Редактировать /etc/rsyslog.d/50-default.conf файл с помощью текстового редактора, например, vi или nano:

$ sudo vi /etc/rsyslog.d/50-default.conf

$ sudo vi /etc/rsyslog.d/50-default.conf

$ sudo nano /etc/rsyslog.d/50-default.conf

$ sudo nano /etc/rsyslog.d/50-default.conf

Необходимо её раскомментировать, и перезапустить оба сервиса:

$ sudo service rsyslog restart

$ sudo service cron restart

$ sudo service rsyslog restart

$ sudo service cron restart

Теперь все сообщения об ошибках в работе планировщика можно будет просматривать в файле

Воспользуйтесь различными командами для поиска нужной информации в логе

Источник

Where are Cron Job errors logged?

As the title states. Where are the crontab errors logged and what is the name of the file? I’m not sure what the name of the file is so I have no clue what to search on with find -iname «file name here» …

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

By default on Ubuntu your cron output should be logged to /var/log/syslog If you would like cron to log to it’s own file you can edit /etc/rsyslog.d/50-default.conf and uncomment the line

and your cron jobs will be logged to /var/log/cron.log

Popular Topics

Questions

Sign up for Infrastructure as a Newsletter.

Working on improving health and education, reducing inequality, and spurring economic growth? We’d like to help.

You get paid; we donate to tech nonprofits.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow – whether you’re running one virtual machine or ten thousand.

Источник

Why is my crontab not working, and how can I troubleshoot it?

You have been directed here because the community is fairly sure that the answer to your question can be found below. If your question is not answered below then the answers will help you gather information that will help the community help you. This information should be edited into your original question.

The answer for ‘Why is my crontab not working, and how can I troubleshoot it?‘ can be seen below. This addresses the cron system with the crontab highlighted.

7 Answers 7

This is a community wiki, if you notice anything incorrect with this answer or have additional information then please edit it.

First, basic terminology:

• cron(8) is the daemon that executes scheduled commands.
• crontab(1) is the program used to modify user crontab(5) files.
• crontab(5) is a per user file that contains instructions for cron(8).

Next, education about cron:

Every user on a system may have their own crontab file. The location of the root and user crontab files are system dependant but they are generally below /var/spool/cron .

There is a system-wide /etc/crontab file, the /etc/cron.d directory may contain crontab fragments which are also read and actioned by cron. Some Linux distributions (eg, Red Hat) also have /etc/cron. which are directories, scripts inside which will be executed every hour/day/week/month, with root privilege.

root can always use the crontab command; regular users may or may not be granted access. When you edit the crontab file with the command crontab -e and save it, crond checks it for basic validity but does not guarantee your crontab file is correctly formed. There is a file called cron.deny which will specify which users cannot use cron. The cron.deny file location is system dependent and can be deleted which will allow all users to use cron.

If the computer is not powered on or crond daemon is not running, and the date/time for a command to run has passed, crond will not catchup and run past queries.

crontab particulars, how to formulate a command:

A crontab command is represented by a single line. You cannot use to extend a command over multiple lines. The hash ( # ) sign represents a comment which means anything on that line is ignored by cron. Leading whitespace and blank lines are ignored.

Be VERY careful when using the percent ( % ) sign in your command. Unless they are escaped % they are converted into newlines and everything after the first non-escaped % is passed to your command on stdin.

There are two formats for crontab files:

System wide /etc/crontab and /etc/cron.d fragments

Notice that the latter requires a user-name. The command will be run as the named user.

The first 5 fields of the line represent the time(s) when the command should be run. You can use numbers or where applicable day/month names in the time specification.

• The fields are separated by spaces or tabs.
• A comma ( , ) is used to specify a list e.g 1,4,6,8 which means run at 1,4,6,8.
• Ranges are specified with a dash ( — ) and may be combined with lists e.g. 1-3,9-12 which means between 1 and 3 then between 9 and 12.
• The / character can be used to introduce a step e.g. 2/5 which means starting at 2 then every 5 (2,7,12,17,22. ). They do not wrap past the end.
• An asterisk ( * ) in a field signifies the entire range for that field (e.g. 0-59 for the minute field).
• Ranges and steps can be combined e.g. */2 signifies starting at the minimum for the relevant field then every 2 e.g. 0 for minutes( 0,2. 58), 1 for months (1,3 . 11) etc.

Debugging cron commands

Check the mail!

By default cron will mail any output from the command to the user it is running the command as. If there is no output there will be no mail. If you want cron to send mail to a different account then you can set the MAILTO environment variable in the crontab file e.g.

Capture the output yourself

You can redirect stdout and stderr to a file. The exact syntax for capturing output may vary depending on what shell cron is using. Here are two examples which save all output to a file at /tmp/mycommand.log :

Look at the logs

Cron logs its actions via syslog, which (depending on your setup) often go to /var/log/cron or /var/log/syslog .

If required you can filter the cron statements with e.g.

Now that we’ve gone over the basics of cron, where the files are and how to use them let’s look at some common problems.

Check that cron is running

If cron isn’t running then your commands won’t be scheduled .

should get you something like

If not restart it

There may be other methods; use what your distro provides.

cron runs your command in a restricted environment.

What environment variables are available is likely to be very limited. Typically, you’ll only get a few variables defined, such as $LOGNAME , $HOME , and $PATH .

Of particular note is the PATH is restricted to /bin:/usr/bin . The vast majority of «my cron script doesn’t work» problems are caused by this restrictive path. If your command is in a different location you can solve this in a couple of ways:

Provide the full path to your command.

Provide a suitable PATH in the crontab file

If your command requires other environment variables you can define them in the crontab file too.

cron runs your command with cwd == $HOME

Regardless of where the program you execute resides on the filesystem, the current working directory of the program when cron runs it will be the user’s home directory. If you access files in your program, you’ll need to take this into account if you use relative paths, or (preferably) just use fully-qualified paths everywhere, and save everyone a whole lot of confusion.

The last command in my crontab doesn’t run

Cron generally requires that commands are terminated with a new line. Edit your crontab; go to the end of the line which contains the last command and insert a new line (press enter).

Check the crontab format

You can’t use a user crontab formatted crontab for /etc/crontab or the fragments in /etc/cron.d and vice versa. A user formatted crontab does not include a username in the 6th position of a row, while a system formatted crontab includes the username and runs the command as that user.

I put a file in /etc/cron. and it doesn’t run

• Check that the filename doesn’t have an extension see run-parts
• Ensure the file has execute permissions.
• Tell the system what to use when executing your script (eg. put #!/bin/sh at top)

If your date is recently changed by a user or system update, timezone or other, then crontab will start behaving erratically and exhibit bizarre bugs, sometimes working, sometimes not. This is crontab’s attempt to try to «do what you want» when the time changes out from underneath it. The «minute» field will become ineffective after the hour is changed. In this scenario, only asterisks would be accepted. Restart cron and try it again without connecting to the internet (so the date doesn’t have a chance to reset to one of the time servers).

Percent signs, again

To emphasise the advice about percent signs, here’s an example of what cron does with them:

will create the

/cron.out file containing the 3 lines

This is particularly intrusive when using the date command. Be sure to escape the percent signs

How to use sudo in cron jobs

when running as a non-root user,

will open the user’s crontab, while

will open the root user’s crontab. It’s not recommended to run sudo commands in a cron job, so if you’re trying to run a sudo command in a user’s cron, try moving that command to root’s cron and remove sudo from the command.

Debian Linux and its derivative (Ubuntu, Mint, etc) have some peculiarities that may prevent your cron jobs from executing; in particular, the files in /etc/cron.d , /etc/cron. must :

• be owned by root
• only be writable by root
• not be writable by group or other users
• have a name without any dots ‘.’ or any other special character but ‘-‘ and ‘_’ .

The last one hurts regularly unsuspecting users; in particular any script in one of these folders named whatever.sh , mycron.py , testfile.pl , etc. will not be executed, ever.

In my experience, this particular point has been by far the most frequent reason for a non-executing cronjob on Debian and derivatives.

See man cron for more details, if necessary.

If your cronjobs stop working, check that your password hasnt expired., since once it has, all cron jobs stop.
There will be messages in /var/log/messages similar to the one below which show issues with authenticating the user:

(username) FAILED to authorize user with PAM (Authentication token is no longer valid; new one required)

Uncommon and irregular schedules

Cron is all things considered a very basic scheduler and the syntax does not easily allow an administrator to formulate slightly more uncommon schedules.

Consider the following job which commonly would be explained to «run command every 5 minutes»:

which does not always run command every 7 minutes.

Remember that the / character can be used to introduce a step but that steps don’t wrap beyond the end of a series e.g. */7 which matches every 7th minute from the minutes 0-59 i.e. 0,7,14,21,28,35,42,49,56 but between one hour and the next there will be only 4 minutes between batches, after 00:56 a new series starts at 01:00 , 01:07 etc. (and batches won’t run on 01:03 , 01:10 , 01:17 etc.).

What to do instead?

Create multiple batches

Rather than a single cron job, create multiple batches that combined result in the desired schedule.

For instance to run a batch every 40 minutes (00:00, 00:40, 01:20, 02:00 etc.) create two batches, one that runs twice on the even hours and second one that runs only the odd hours:

Run your batches less frequently

Rather than running your batch every 7 minutes, which is a difficult schedule to break down in multiple batches, simply run it every 10 minutes instead.

Start your batches more frequently (but prevent multiple batches from running concurrently)

Many odd schedules evolve because the batch runtimes increase/fluctuate and then the batches get scheduled with a bit of additional safety margin to prevent subsequent runs of the same batch from overlapping and running concurrently.

Instead, think differently and create a cronjob that will fail gracefully when a previous run has not finished yet, but which will run otherwise. See this Q&A:

That will almost immediately start a new run once the previous run of /usr/local/bin/frequent_cron_job has completed.

Start your batches more frequently (but exit gracefully when the conditions are not right)

Since cron syntax is limited you may decide to place more complex conditions and logic in the batch job itself (or in a wrapper script around the existing batch job). That allows you to utilize the advanced capabilities of your favorite scripting languages, to comment your code and will prevent hard-to-read constructs in the crontab entry itself.

In bash the seven-minute-job would then look something like something like:

Which you can then safely (attempt) to run every minute:

A different, but similar problem would to schedule a batch to run on the first Monday of every month (or the second Wednesday) etc. Simply schedule the batch to run every Monday and exit when date is neither between the 1 st or 7 th and the day of the week is not Monday.

Which you can then safely (attempt) to run every Monday:

Don’t use cron

If your needs are complex you might consider using a more advanced product that is designed to run complex schedules (distributed over multiple servers) and that supports triggers, job dependencies, error handling, retries and retry monitoring etc. The industry jargon would be «enterprise» job scheduling and/or «workload automation».

PHP-specific

If you have some cron job like:

And in case of errors expect, that they will be sent to you, but they not — check this.

PHP by default not sending errors to STDOUT. @see https://bugs.php.net/bug.php?id=22839

To fix this, add in cli`s php.ini or in your line (or in your’s bash wrapper for PHP) these:

• —define display_startup_errors=1
• —define display_errors=’stderr’

1st setting will allow you to have fatals like ‘Memory oops’ and 2nd — to redirect them all to STDERR. Only after you can sleep well as all will be sent to your root’s mail instead of just logged.

Adding my answer from here for completeness, and adding another potentially helpful resource:

The cron user has a different $PATH than you do:

A frequent problem users make with crontab entries is that they forget that cron runs in a different environment than they do as a logged-in user. For example, a user creates a program or script in his $HOME directory, and enters the following command to run it:

The command runs perfectly from his command line. The user then adds that command to his crontab , but finds this does not work:

The reason for the failure in this case is that ./ is a different location for the cron user than it is for the logged-in user. That is, the environment is different! The PATH is part of the environment , and it is usually different for the cron user. Complicating this issue is that the environment for cron is not the same for all *nix distributions, and there are multiple versions of cron

A simple solution to this particular problem is to give the cron user a complete path specification in the crontab entry:

What is the cron user’s environment ?

In some instances, we may need to know the complete environment specification for cron on our system (or we may just be curious). What is the environment for the cron user, and how is it different from ours? Further, we may need to know the environment for another cron user — root for example. what is the root user’s environment using cron ? One way to learn this is to ask cron to tell us:

Create a shell script in your home directory (

/ ) as follows (or with the editor of your choice):

1. Enter the following in the editor, after adjusting for your system/user:

1. Save the file, exit the editor and set file permissions as executable.

1. Run the script you just created, and review the output in /home/you/envtst.sh.out . This output will show your current environment as the $USER you’re logged in as:

1. Open your crontab for editing:

1. Enter the following line at the bottom of your crontab :

ANSWER: The output file /home/you/envtst.sh.out will contain a listing of the environment for the «root cron user». Once you know that, adjust your crontab entry accordingly.

I can’t specify the schedule I need in my crontab entry:

The schedule entry for crontab is of course defined in man crontab , and you should read this. However, reading man crontab , and understanding the schedule are two different things. And trial-and-error on a schedule specification can become very tedious. Fortunately, there is a resource that can help: the crontab guru.. Enter your schedule specification, and it will explain the schedule in plain English language.

Finally, and at risk of being redundant with one of the other answers here, do not get trapped into thinking that you are limited to a single crontab entry because you have one job to schedule. You are free to use as many crontab entries as you need to get the schedule you need.

Источник

Как пользователь Linux, вы, вероятно, уже знакомы с crontab.

Вы можете автоматизировать задачи, запуская команды и скрипты по заданному расписанию.

Хотите автоматически создавать резервные копии?

Crontab – вам в этом поможет.

Здесь я не буду углубляться в использование crontab.

Я сосредоточусь на том, чтобы показать вам различные способы проверки логов crontab.

Это поможет выяснить, выполнились ли ваши cronjobs по расписанию или нет.

Метод 1: Проверим syslog на наличие записей crontab

Согласно иерархии каталогов Linux, в каталоге /var/log в Linux хранятся логи системы, служб и запущенных приложений.

Хотя журналы cron также находятся в этом каталоге, стандартного файла для этих журналов не существует. Разные дистрибутивы хранят их в разных файлах.

🐧 Руководство для начинающих по системным логам в системах Linux

В дистрибутивах на базе Debian файл /var/log/syslog содержит логи выполнения заданий cron, и к нему следует обращаться в случае, если задания cron не работают:

cat /var/log/syslog | grep -w 'cron'

После выполнения вышеуказанной команды вы увидите в терминале список всех заданий cron.

Команда grep отфильтрует сообщения, связанные с cron, от остальных.

Для дистрибутивов на базе RedHat логи cron находятся в специальном файле /var/log/cron.

В обоих случаях для доступа к журналам вам, вероятно, потребуется указать ключевое слово sudo или использовать учетную запись root.

Метод 2: Использование пользовательского файла логов (рекомендуется)

Рекомендуется использовать отдельный пользовательский файл для регистрации заданий cron.

Для этого вы можете настроить ‘rsyslog’ для пересылки журналов cron.

Rsyslog – это служба Linux, которая имеет функции, аналогичные протоколированию Syslog.

Просто создайте файл cron.log в каталоге /etc/rsyslog.d:

touch /var/log/cron.log

Теперь откройте для редактирования файл /etc/rsyslog.d/50-default.conf:

nano /etc/rsyslog.d/50-default.conf 

и найдите строку, начинающуюся с #cron.* и удалите # в начале строки.

Чтобы изменения заработали, сохраните и закройте этот файл и, наконец, перезапустите службу rsyslog и проверьте ее состояние:

sudo systemctl restart rsyslog

sudo systemctl status rsyslog

Статус службы должен быть выделен как active (running).

Метод 3: Использование специальных служб, такиъ как Cronitor, для мониторинга заданий cron

Cronitor – это служба, которую можно развернуть для мониторинга любого типа заданий cron.

Многие версии cron начинают вести журнал при выполнении запланированного задания или при возникновении проблем с crontab.

Однако, выход из задания cron или статус его завершения не регистрируется.

Здесь на помощь приходит Cronitor, который работает идеально.

Это комплексное решение для всех ваших потребностей в crontab.

🐧 Обзор средств мониторинга и управления логами с открытым исходным кодом для Linux

Он собирает логи, метрики и статус всех заданий и создает мгновенные предупреждения о сбое или неудачном запуске задания.

Все это можно увидеть через веб-интерфейс.

Для Cronitor или CronitorCLI, установленных на Kubernetes, журналы могут быть записаны в объеме до 100 МБ за одно выполнение.

Другие инструменты и сервисы мониторинга, такие как Better Uptime, также предоставляют возможность автоматического мониторинга заданий cron.

🐧 Как использовать journalctl для анализа логов на Linux

Заключение

Системные журналы очень важны для поиска и диагностики проблем, связанных с системой, и журналы cron не являются исключением.

Syslog хранит журналы, связанные с crontab, однако рекомендуется иметь специальный файл журнала для cron.

Также поможет веб-сервис, например, Cronitor, Better Uptime или Uptime Robot.

см. также:

• 🐧 Как запустить несколько команд в одном задании Cron
• 🐍 Планирование выполнения скриптов Python с помощью Crontab
• ⏲️ Запуск задания cron каждые 12 часов (дважды в день)
• 🐧 Запуск Cron каждые 30 секунд
• ⏲️ Как просмотреть или составить список заданий Cron на Linux
• ☸️ Как создать job Kubernetes из работы cron job
• 😿 Как перечислить задания Cron в Linux

How to easily log all output from any executable

…what I need right now is to learn how to

1. Find log files about cron jobs
2. Configure what gets logged

Summary

Put this magic line at the top of any script, including inside your scripts or wrapper scripts which get called as cron jobs, in order to have all stdout and stderr output thereafter get automatically logged!

exec > >(tee -a "$HOME/cronjob_logs/my_log.log") 2>&1

Details

Cron jobs are just scripts that get called at a fixed time or interval by a scheduler. They log only what their executables tell them to log.

But, what you can do is edit the call command as listed in your crontab -e file in such a way that it causes all output to get logged when you call it.

There are a variety of approaches, such as directing it to a file at call time, like this:

# `crontab -e` entry which gets called every day at 2am
0 2 * * * some_cmd > $HOME/cronjob_logs/some_cmd.log 2>&1

But, what if I want it to automatically log when I call it normally to test it, too? I’d like it to log every time I call some_cmd, and I’d still like it to show all output to the screen as well so that when I call the script manually to test it I still see all output! The best way I have come up with is this. It looks really weird, but it’s very powerful! Simply add these lines to the top of any script and all stdout or stderr output thereafter will get automatically logged to the specified file, named after the script you are running itself! So, if your script is called some_cmd, then all output will get logged to $HOME/cronjob_logs/some_cmd.log:

# See my ans: https://stackoverflow.com/a/60157372/4561887
FULL_PATH_TO_SCRIPT="$(realpath "${BASH_SOURCE[-1]}")"
SCRIPT_DIRECTORY="$(dirname "$FULL_PATH_TO_SCRIPT")"
SCRIPT_FILENAME="$(basename "$FULL_PATH_TO_SCRIPT")"
# See:
# 1. https://stackoverflow.com/a/49514467/4561887 and 
# 2. https://superuser.com/a/569315/425838
mkdir -p "$HOME/cronjob_logs"
exec > >(tee -a "$HOME/cronjob_logs/${SCRIPT_FILENAME}.log") 2>&1 # <=MAGIC CMD

Here are the relevant references from above to understand it all:

1. [my answer] How to obtain the full file path, full directory, and base filename of any script being run OR sourced…
2. Explain the bash command exec > >(tee $LOG_FILE) 2>&1 — «process substitution»
3. How can I view results of my cron jobs? — shows something similar to the «process substitution» above

Let’s see this in full context, with a little beautification and a bunch of explanatory comments, where I am wrapping some_executable (any executable: bash, C, C++, Python, whatever) with a some_executable.sh bash wrapper to enable logging:

some_executable.sh:

# -------------- cron job automatic logger code START --------------

# See my ans: https://stackoverflow.com/a/60157372/4561887
FULL_PATH_TO_SCRIPT="$(realpath "${BASH_SOURCE[-1]}")"
SCRIPT_DIRECTORY="$(dirname "$FULL_PATH_TO_SCRIPT")"
SCRIPT_FILENAME="$(basename "$FULL_PATH_TO_SCRIPT")"

# Automatically log the output of this script to a file!
begin_logging() {
    mkdir -p ~/cronjob_logs

    # Redirect all future prints in this script from this call-point forward to
    # both the screen and a log file!
    #
    # This is about as magic as it gets! This line uses `exec` + bash "process
    # substitution" to redirect all future print statements in this script
    # after this line from `stdout` to the `tee` command used below, instead.
    # This way, they get printed to the screen *and* to the specified log file
    # here! The `2>&1` part redirects `stderr` to `stdout` as well, so that
    # `stderr` output gets logged into the file too.
    # See:
    # 1. *****+++ https://stackoverflow.com/a/49514467/4561887 -
    #    shows `exec > >(tee $LOG_FILE) 2>&1`
    # 1. https://superuser.com/a/569315/425838 - shows `exec &>>` (similar)
    exec > >(tee -a "$HOME/cronjob_logs/${SCRIPT_FILENAME}.log") 2>&1

    echo ""
    echo "====================================================================="
    echo "Running cronjob "$FULL_PATH_TO_SCRIPT""
    echo "on $(date)."
    echo "Cmd:  $0 $@"
    echo "====================================================================="
}

# --------------- cron job automatic logger code END ---------------
# THE REST OF THE SCRIPT GOES BELOW THIS POINT.
# ------------------------------------------------------------------

main() {
    some_executable
    echo "= DONE."
    echo ""
}

# ------------------------------------------------------------------------------
# main program entry point
# ------------------------------------------------------------------------------
begin_logging "$@"
time main "$@"

There you have it! Now isn’t that pretty and nice!? Just put whatever you want in the main() function, and all of the output there gets automatically logged into $HOME/cronjob_logs/some_executable.sh.log, whether you call this wrapper manually or whether it is called by a cron job! And, since the tee command is used, all output goes to the terminal too when you run it manually so you can see it live as well.

Each log entry gets prefixed with a nice header too.

Here is an example log entry:

=========================================================================================
Running cronjob "/home/gabriel/GS/dev/eRCaGuy_dotfiles/useful_scripts/cronjobs/repo_git_pull_latest.sh"
on Wed Aug 10 02:00:01 MST 2022.
Cmd:  REMOTE_NAME="origin" MAIN_BRANCH_NAME="main" PATH_TO_REPO="/home/gabriel/GS/dev/random_repo" "/home/gabriel/GS/dev/eRCaGuy_dotfiles/useful_scripts/cronjobs/repo_git_pull_latest.sh" 
=========================================================================================
= Checking to see if the git server is up and running (will time out after 40s)...
= git server is NOT available; exiting now

real    0m40.020s
user    0m0.012s
sys 0m0.003s

You can find and borrow this whole script from my eRCaGuy_dotfiles repo here: repo_git_pull_latest.sh.

See also my cronjob readme and notes here: https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles/tree/master/useful_scripts/cronjobs

In this post you can find information how to deal with problems related to CRON jobs:

• how to find logs
• identify errors
• (CRON) info (No MTA installed, discarding output)

If you need more info about CRON scheduler:

• Start and stop application from terminal in Linux Mint 19 / Ubuntu 18.04 — example for beginners
• Job automation in Linux Mint for beginners 2019 — advanced examples

Step #1: Where is the CRON log in Linux Mint

By default the logs in Ubuntu and Linux Mint from CRON execution can be found in: /var/log/syslog. In this log file there are many different log messages so best way to extract CRON related messages is by:

grep CRON /var/log/syslog

output:

Mar 15 09:00:01 vanx-machine CRON[7205]: (root) CMD (timeshift --check --scripted)
Mar 15 09:12:01 vanx-machine CRON[8128]: (vanx) CMD (/home/vanx/cron/seoanalyzer.sh)
Mar 15 09:17:01 vanx-machine CRON[8638]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar 15 09:30:01 vanx-machine CRON[9508]: (vanx) CMD (DISPLAY=:0 /home/vanx/cron/weekly.sh)
Mar 15 09:32:29 vanx-machine CRON[9507]: (CRON) info (No MTA installed, discarding output)

This command will output all recent executions. If the result is empty most probably nothing was scheduled. Then you can check by:

crontab -e

Step #2: Reading the logs

Successful execution is logged as:

Mar 15 09:12:01 vanx-machine CRON[8128]: (vanx) CMD (/home/user/cron/seoanalyzer.sh)

You can find next information:

• execution date — 15th of March
• execution time — 09:12:01
• machine — vanx-machine
• user — vanx
• command — /home/user/cron/seoanalyzer.sh

In case of errors you will see something like(unless you have MTA installed):

Mar 15 09:32:29 vanx-machine CRON[9507]: (CRON) info (No MTA installed, discarding output)

This error will be covered in next section.

Step #3: (CRON) info (No MTA installed, discarding output)

From the output above we notice this message:

(CRON) info (No MTA installed, discarding output)

The message indicates two things:

• the last job execution failed for some reason
• MTA (Mail Transfer Agent) is not installed (which is the case for Linux Mint and Ubuntu)

The solution of No MTA installed, discarding output is:

1. installation of postfix — as local setup — by using next commands:

sudo aptitude install postfix

1. Choose Local during setup
2. Select name — i.e. — your user name
3. Access the logs(you will need to rerun the failing job):

sudo tail -f /var/mail/vanx

By last command you can find detailed information about the errors and output of the CRON scripts if any.

Example:

  File "/home/vanx/Software/Tensorflow/environments/venv36/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 966, in find_element
    'value': value})['value']
  File "/home/vanx/Software/Tensorflow/environments/venv36/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "/home/vanx/Software/Tensorflow/environments/venv36/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"galaxyIframe"}

Step #4: Redirect CRON job output(no installation of MTA)

If you:

• can’t install MTA
• don’t like the idea of MTA
• installation of additional software on your machine
• or want to analyse single CRON job

then you can redirect the output of your CRON job by command next steps:

1. Edit the CRON schedule file

crontab -e

1. Find the command for which you like to redirect output: 0 3 * * * /cmd/to/run.sh
2. Redirect the output by this command

0 3 * * * /cmd/to/run.sh >> /var/log/cmd_to_run_log_file.log

1. Run the command again and check what happened by:

sudo tail -f -n 50 /var/log/cmd_to_run_log_file.log

Where command tail will product 50 last lines in the given file.

Note 1: Use use if the issues are related to root CRON — sudo crontab -e
Note 2: You can redirect output by 0 3 * * * cmd >> /var/log/cmd_to_run_log_file.log 2>&1

The cron job scheduler has been bundled with unix and linux operating systems for decades. It’s widely used and usually very reliable.
This guide will walk you through locating cron jobs on your server, validating their schedules, and debugging common problems.

In this guide

1. The common reasons cron jobs fail
2. What to do if your cron job isn’t running when expected
3. What to do if your cron job fails unexpectedly
4. What to do if nothing else works

The common reasons cron jobs fail

Cron jobs can fail for an infinite variety of reasons but at Cronitor we have observed common failure patterns that
can provide a useful starting point for your debugging process. In the rest of this guide, we will dive into these
common failures with suggestions and solutions.

If you’re adding a new cron job and it is not working…

• Schedule errors are easy to make

  Cron job schedule expressions are quirky and difficult to write. If your job didn’t run when you expected it to,
  the easiest thing to rule out is a mistake with the cron expression.

• Cron has subtle environment differences

  A common experience is to have a job that works flawlessly when run at the command line but fails whenever
  it’s run by cron. When this happens, eliminate these common failures:

  1. The command has an unresovable relative path like ../scripts. (Try an absolute path)
  2. The job uses environment variables. (Cron does not load .bashrc and similar files)
  3. The command uses advanced bash features (cron uses /bin/sh by default)

  Tip:Our free software, CronitorCLI, includes a shell command to test run any job the way cron does.

• There is a problem with permissions

  Invalid permissions can cause your cron jobs to fail in at least 3 ways. This guide will cover them all.

If your cron job stopped working suddenly…

• System resources have been depleted

  The most stable cron job is no match for a disk that is full or an OS that can’t spawn new threads. Check all the usual graphs to rule this out.

• You’ve reached an inflection point

  Cron jobs are often used for batch processing and other data-intensive tasks that can reveal the constraints of your stack. Jobs often work fine until your data size grows to a point where
  queries start timing-out or file transfers are too slow.

• Infrastructure drift occurs

  When app configuration and code changes are deployed it can be easy to overlook the cron jobs on each server. This causes infrastructure drift where hosts are retired or credentials change that break the forgotten cron jobs.

• Jobs have begun to overlap themselves

  Cron is a very simple scheduler that starts a job at the scheduled time, even if the previous invocation is still running. A small slow-down can lead to a pile-up of overlapped jobs sucking up available resources.

• A bug has been introduced

  Sometimes a failure has nothing to do with cron. It can be difficult to thoroughly test cron jobs in a development environment and a bug might exist only in production.

What to do if your cron job doesn’t run when expected

You may find that you have scheduled a cron job that is just not running when expected. Follow these steps to locate your scheduled job and diagnose the problem.

1. Locate the scheduled job

   Tip:If you know where your job is scheduled, skip this step

   Cron jobs are run by a system daemon called crond that watches several locations for scheduled jobs. The first step to understanding why your job didn’t start when expected
   is to find where your job is scheduled.

   Search manually for cron jobs on your server

   • Check your user crontab with crontab -l

     dev01: ~ $ crontab -l
     # Edit this file to introduce tasks to be run by cron.
     # m h  dom mon dow   command
     5 4 * * *      /var/cronitor/bin/database-backup.sh

   • Jobs are commonly created by adding a crontab file in /etc/cron.d/
   • System-level cron jobs can also be added as a line in /etc/crontab
   • Sometimes for easy scheduling, jobs are added to /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/
     or /etc/cron.monthly/
   • It’s possible that the job was created in the crontab of another user. Go through each user’s crontab using crontab -u username -l

   Or, scan for cron jobs automatically with CronitorCLI

   • Install CronitorCLI for free

   • Run cronitor list to scan your system for cron jobs:
     

     

   If you can’t find your job but believe it was previously scheduled double check that you are on the correct server.

2. Validate your job schedule

   Once you have found your job, verify that it’s scheduled correctly. Cron schedules are commonly used because they are expressive and powerful, but like regular expressions can sometimes be difficult to read.
   We suggest using Crontab Guru to validate your schedule.

   • Paste the schedule expression from your crontab into the text field on Crontab Guru
   • Verify that the plaintext translation of your schedule is correct, and that the next scheduled execution times match your expectations
   • Check that the effective server timezone matches your expectation. In addition to checking system time using date,
     check your crontab file for TZ or CRON_TZ timezone declarations that would override system settings.
     For example, CRON_TZ=America/New_York

3. Check your permissions

   Invalid permissions can cause your cron jobs to fail in at least 3 ways:

   1. Jobs added as files in a /etc/cron.*/ directory must be owned by root. Files owned by other users will be ignored and you may see a message similar to
      WRONG FILE OWNER in your syslog.
   2. The command must be executable by the user that cron is running your job as. For example if your ubuntu user crontab invokes a script like
      database-backup.sh, ubuntu must have permission to execute the script. The most direct way is to ensure that the ubuntu
      user owns the file and then ensure execute permissions are available using chmod +x database-backup.sh.
   3. The user account must be allowed to use cron. First, if a /etc/cron.allow file exists, the user must be listed. Separately,
      the user cannot be in a /etc/cron.deny list.

   Related to the permissions problem, ensure that if your command string contains a % that it is escaped with a backslash.

4. Check that your cron job is running by finding the attempted execution in syslog

   When cron attempts to run a command, it logs it in syslog.
   By grepping syslog for the name of the command you found in a crontab file you can validate that your job is scheduled correctly and cron is running.

   • Begin by grepping for the command in /var/log/syslog (You will probably need root or sudo access.)

     dev01: ~ $ grep database-backup.sh /var/log/syslog
     Aug  5 4:05:01 dev01 CRON[2128]: (ubuntu) CMD (/var/cronitor/bin/database-backup.sh)

   • If you can’t find your command in the syslog it could be that the log has been rotated or cleared since your job ran. If possible, rule that out by updating the job to run every minute by changing its schedule to * * * * *.
   • If your command doesn’t appear as an entry in syslog within 2 minutes the problem could be with the underlying cron daemon known as crond. Rule this out quickly by verifying that cron is running by looking up its process ID. If cron is not running no process ID will be returned.

     dev01: ~ $ pgrep cron
     323

   • If you’ve located your job in a crontab file but persistently cannot find it referenced in syslog, double check that crond has correctly loaded your crontab file. The easiest way to do this is to force
     a reparse of your crontab by running EDITOR=true crontab -e from your command prompt. If everything is up to date you will see a message like No modification made. Any other
     message indicates that your crontab file was not reloaded after a previous update but has now been updated. This will also ensure that your crontab file is free of syntax errors.

If you can see in syslog that your job was scheduled and attempted to run correctly but still did not produce the expected result you can assume there is a problem with the command you are trying to run.

What to do if your cron job fails unexpectedly

Your cron jobs, like every other part of your system, will eventually fail. This section will walk you through testing your job and spotting common failures.

1. Test run your command like cron does

   When cron runs your command the environment is different from your normal command prompt in subtle but important ways. The first step to troubleshooting is to simulate the cron environment and
   run your command in an interactive shell.

   Run any command like cron does with CronitorCLI

   • Install CronitorCLI for free

   • To force a scheduled cron job to run immediately, use cronitor select to scan your system and present a list of jobs to choose from.
   • To simulate running any command the way cron does, use cronitor shell:
     

     

   Or, manually test run a command like cron does

   • If you are parsing a file in /etc/cron.d/ or /etc/crontab each line is allowed to have an effective «run as» username
     after the schedule and before the command itself. If this applies to your job, or if your job is in another user’s crontab, begin by opening a bash prompt as that user sudo -u username bash
   • By default, cron will run your command using /bin/sh, not the bash or zsh prompt you are familiar with. Double check your crontab file for an optional
     SHELL=/bin/bash declaration. If using the default /bin/sh shell, certain features that work in bash like [[command]] syntax will cause syntax errors under cron.
   • Unlike your interactive shell, cron doesn’t load your bashrc or bash_profile so any environment variables defined there are unavailable in your cron jobs. This is true even if you have a SHELL=/bin/bash declaration. To simulate this, create a command prompt with a clean environment.

     dev01: ~ $ env -i /bin/sh
     $

   • By default, cron will run commands with your home directory as the current working directory. To ensure you are running the command like cron does, run cd ~ at your prompt.
   • Paste the command to run (everything after the schedule or declared username) into the command prompt. If crontab is unable to run your command, this should fail too and will hopefully contain a useful error message.
     Common errors include invalid permissions, command not found, and command line syntax errors.
   • If no useful error message is available, double check any application logs your job is expected to produce, and ensure that you are not redirecting log and error messages.
     In linux, command >> /path/to/file will redirect console log messages to the specified file and command >> /path/to/file 2>&1 will redirect both the console log and error messages.
     Determine if your command has a verbose output or debug log flag that can be added to see additional details at runtime.
   • Ideally, if your job is failing under cron it will fail here too and you will see a useful error message, giving you a chance to modify and debug as needed. Common errors include file not found, misconfigured permissions, and commandline syntax errors.

2. Check for overlapping jobs

   At Cronitor our data shows that runtime durations increase over time for a large percentage of cron jobs. As your dataset and user base grows it’s normal to find yourself in a situation
   where cron starts an instance of your job before the previous one has finished. Depending on the nature of your job this might not be a problem, but several undesired side effects are possible:

   • Unexpected server or database load could impact other users
   • Locking of shared resources could cause deadlocks and prevent your jobs from ever completing successfully
   • Creation of an unanticipated race condition that might result in records being processed multiple times, amplifying server load and possibly impacting customers

   To verify if any instances of a job are running on your server presently, grep your process list. In this example, 3 job invocations are running simultaneously:

   dev01: ~ $ ps aux | grep database-backup.sh
   ubuntu           1343   0.0  0.1  2585948  12004   ??  S    31Jul18   1:04.15 /var/cronitor/bin/database-backup.sh
   ubuntu           3659   0.0  0.1  2544664    952   ??  S     1Aug18   0:34.35 /var/cronitor/bin/database-backup.sh
   ubuntu           7309   0.0  0.1  2544664   8012   ??  S     2Aug18   0:18.01 /var/cronitor/bin/database-backup.sh

   To quickly recover from this, first kill the overlapping jobs and then watch closely when your command is next scheduled to run. It’s possible that a one time failure cascaded into several overlapping instances.
   If it becomes clear that the job often takes longer than the interval between job invocations you may need to take additional steps, e.g.:

   • Increase the duration between invocations of your job. For example if your job runs every minute now, consider running it every other minute.
   • Use a tool like flock to ensure that only a single instance of your command is running at any given time. Using flock is easy. After installing
     from apt-get or yum you only need to prefix the command in your crontab:

   dev01: ~ $ crontab -l
   # Edit this file to introduce tasks to be run by cron.
   # m h  dom mon dow   command
   * * * * *      flock -w 0 /var/cronitor/bin/database-backup.sh

   Read more about flock from the man page.

What to do if nothing else works

Here are a few things you can try if you’ve followed this guide and find that your job works flawlessly when run from the cron-like command prompt but fails to complete successful under crontab.

• First get the most basic cron job working with a command like date >> /tmp/cronlog. This command will simply echo the execution time to the log file each time it runs. Schedule this to run every minute and tail the logfile for results.

• If your basic command works, replace it with your command. As a sanity check, verify if it works.

• If your command works by invoking a runtime like python some-command.py perform a few checks to determine that the runtime version and environment is correct. Each language runtime has quirks
  that can cause unexpected behavior under crontab.

  • For python you might find that your web app is using a virtual environment you need to invoke in your crontab.
  • For node a common problem is falling back to a much older version bundled with the distribution.
  • When using php you might run into the issue that custom settings or extensions that work in your web app
    are not working under cron or commandline because a different php.ini is loaded.

  If you are using a runtime to invoke your command double check that it’s pointing to the expected version from within the cron environment.

• If nothing else works, restart the cron daemon and hope it helps, stranger things have happened. The way to do this varies from distro to distro so it’s best to ask google.

---

Cronitor automatically finds and monitors cron jobs.  Complete your setup in minutes.

Get Started Free