Cron daemon error

As this is becoming a canonical for troubleshooting cron issues, allow me to add one specific but rather complex issue: If you are attempting to run a GUI program from cron, you are probably Doing It Wrong.

A common symptom is receiving error messages about DISPLAY being unset, or the cron job’s process being unable to access the display.

In brief, this means that the program you are trying to run is attempting to render something on an X11 (or Wayland etc) display, and failing, because cron is not attached to a graphical environment, or in fact any kind of input/output facility at all, beyond being able to read and write files, and send email if the system is configured to allow that.

For the purposes of «I’m unable to run my graphical cron job», let’s just point out in broad strokes three common scenarios for this problem.
Probably identify the case you are trying to implement, and search for related questions about that particular scenario to learn more, and find actual solutions with actual code.

1. If you are trying to develop an interactive program which communicates with a user, you want to rethink your approach. A common, but nontrivial, arrangement is to split the program in two: A back-end service which can run from cron, but which does not have any user-visible interactive facilities, and a front-end client which the user runs from their GUI when they want to communicate with the back-end service.

   Probably your user client should simply be added to the user(s)’ GUI startup script if it needs to be, or they want to, run automatically when they log in.

   I suppose the back-end service could be started from cron, but if it requires a GUI to be useful, maybe start it from the X11 server’s startup scripts instead; and if not, probably run it from a regular startup script (systemd these days, or /etc/rc.local or a similar system startup directory more traditionally).¹

2. If you are trying to run a GUI program without interacting with a real user ², you may be able to set up a «headless» X11 server ³ and run a cron job which starts up that server, runs your job, and quits.

   Probably your job should simply run a suitable X11 server from cron (separate from any interactive X11 server which manages the actual physical display(s) and attached graphics card(s) and keyboard(s) available to the system), and pass it a configuration which runs the client(s) you want to run once it’s up and running. (See also the next point for some practical considerations.)

3. You are running a computer for the sole purpose of displaying a specific application in a GUI, and you want to start that application when the computer is booted.

   Probably your startup scripts should simply run the GUI (X11 or whatever) and hook into its startup script to also run the client program once the GUI is up and running. In other words, you don’t need cron here; just configure the startup scripts to run the desktop GUI, and configure the desktop GUI to run your application as part of the (presumably automatic, guest?) login sequence.⁴

There are ways to run X11 programs on the system’s primary display (DISPLAY=:0.0) but doing that from a cron job is often problematic, as that display is usually reserved for actual interactive use by the first user who logs in and starts a graphical desktop. On a single-user system, you might be able to live with the side effects if that user is also you, but this tends to have inconvenient consequences and scale very poorly.

An additional complication is deciding which user to run the cron job as. A shared system resource like a back-end service can and probably should be run by root (though ideally have a dedicated system account which it switches into once it has acquired access to any privileged resources it needs) but anything involving a GUI should definitely not be run as root at any point.

A related, but distinct problem is to interact in any meaningful way with the user. If you can identify the user’s active session (to the extent that this is even well-defined in the first place), how do you grab their attention without interfering with whatever else they are in the middle of? But more fundamentally, how do you even find them? If they are not logged in at all, what do you do then? If they are, how do you determine that they are active and available? If they are logged in more than once, which terminal are they using, and is it safe to interrupt that session? Similarly, if they are logged in to the GUI, they might miss a window you spring up on the local console, if they are actually logged in remotely via VNC or a remote X11 server.

As a further aside: On dedicated servers (web hosting services, supercomputing clusters, etc) you might even be breaking the terms of service of the hosting company or institution if you install an interactive graphical desktop you can connect to from the outside world, or even at all.

¹
The @reboot hook in cron is a convenience for regular users who don’t have any other facility for running something when the system comes up, but it’s just inconvenient and obscure to hide something there if you are root anyway and have complete control over the system. Use the system facilities to launch system services.

²
A common use case is running a web browser which needs to run a full GUI client, but which is being controlled programmatically and which doesn’t really need to display anything anywhere, for example to scrape sites which use Javascript and thus require a full graphical browser to render the information you want to extract.

Another is poorly designed scientific or office software which was not written for batch use, and thus requires a GUI even when you just want to run a batch job and then immediately quit without any actual need to display anything anywhere.

(In the latter case, probably review the documentation to check if there isn’t a --batch or --noninteractive or --headless or --script or --eval option or similar to run the tool without the GUI, or perhaps a separate utility for noninteractive use.)

³
Xvfb is the de facto standard solution; it runs a «virtual framebuffer» where the computer can spit out pixels as if to a display, but which isn’t actually connected to any display hardware.

⁴
There are several options here.
The absolutely simplest is to set up the system to automatically log in a specific user at startup without a password prompt, and configure that user’s desktop environment (Gnome or KDE or XFCE or what have you) to run your script from its «Startup Items» or «Login Actions» or «Autostart» or whatever the facility might be called. If you need more control over the environment, maybe run bare X11 without a desktop environment or window manager at all, and just run your script instead. Or in some cases, maybe replace the X11 login manager («greeter») with something custom built.

The X11 stack is quite modular, and there are several hooks in various layers where you could run a script either as part of a standard startup process, or one which completely replaces a standard layer. These things tend to differ somewhat between distros and implementations, and over time, so this answer is necessarily vague and incomplete around these matters. Again, probably try to find an existing question about how to do things for your specific platform (Ubuntu, Raspbian, Gnome, KDE, what?) and scenario. For simple scenarios, perhaps see Ubuntu — run bash script on startup with visible terminal

The cron job scheduler has been bundled with unix and linux operating systems for decades. It’s widely used and usually very reliable.
This guide will walk you through locating cron jobs on your server, validating their schedules, and debugging common problems.

In this guide

1. The common reasons cron jobs fail
2. What to do if your cron job isn’t running when expected
3. What to do if your cron job fails unexpectedly
4. What to do if nothing else works

The common reasons cron jobs fail

Cron jobs can fail for an infinite variety of reasons but at Cronitor we have observed common failure patterns that
can provide a useful starting point for your debugging process. In the rest of this guide, we will dive into these
common failures with suggestions and solutions.

If you’re adding a new cron job and it is not working…

• Schedule errors are easy to make

  Cron job schedule expressions are quirky and difficult to write. If your job didn’t run when you expected it to,
  the easiest thing to rule out is a mistake with the cron expression.

• Cron has subtle environment differences

  A common experience is to have a job that works flawlessly when run at the command line but fails whenever
  it’s run by cron. When this happens, eliminate these common failures:

  1. The command has an unresovable relative path like ../scripts. (Try an absolute path)
  2. The job uses environment variables. (Cron does not load .bashrc and similar files)
  3. The command uses advanced bash features (cron uses /bin/sh by default)

  Tip:Our free software, CronitorCLI, includes a shell command to test run any job the way cron does.

• There is a problem with permissions

  Invalid permissions can cause your cron jobs to fail in at least 3 ways. This guide will cover them all.

If your cron job stopped working suddenly…

• System resources have been depleted

  The most stable cron job is no match for a disk that is full or an OS that can’t spawn new threads. Check all the usual graphs to rule this out.

• You’ve reached an inflection point

  Cron jobs are often used for batch processing and other data-intensive tasks that can reveal the constraints of your stack. Jobs often work fine until your data size grows to a point where
  queries start timing-out or file transfers are too slow.

• Infrastructure drift occurs

  When app configuration and code changes are deployed it can be easy to overlook the cron jobs on each server. This causes infrastructure drift where hosts are retired or credentials change that break the forgotten cron jobs.

• Jobs have begun to overlap themselves

  Cron is a very simple scheduler that starts a job at the scheduled time, even if the previous invocation is still running. A small slow-down can lead to a pile-up of overlapped jobs sucking up available resources.

• A bug has been introduced

  Sometimes a failure has nothing to do with cron. It can be difficult to thoroughly test cron jobs in a development environment and a bug might exist only in production.

What to do if your cron job doesn’t run when expected

You may find that you have scheduled a cron job that is just not running when expected. Follow these steps to locate your scheduled job and diagnose the problem.

1. Locate the scheduled job

   Tip:If you know where your job is scheduled, skip this step

   Cron jobs are run by a system daemon called crond that watches several locations for scheduled jobs. The first step to understanding why your job didn’t start when expected
   is to find where your job is scheduled.

   Search manually for cron jobs on your server

   • Check your user crontab with crontab -l

     dev01: ~ $ crontab -l
     # Edit this file to introduce tasks to be run by cron.
     # m h  dom mon dow   command
     5 4 * * *      /var/cronitor/bin/database-backup.sh

   • Jobs are commonly created by adding a crontab file in /etc/cron.d/
   • System-level cron jobs can also be added as a line in /etc/crontab
   • Sometimes for easy scheduling, jobs are added to /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/
     or /etc/cron.monthly/
   • It’s possible that the job was created in the crontab of another user. Go through each user’s crontab using crontab -u username -l

   Or, scan for cron jobs automatically with CronitorCLI

   • Install CronitorCLI for free

   • Run cronitor list to scan your system for cron jobs:
     

     

   If you can’t find your job but believe it was previously scheduled double check that you are on the correct server.

2. Validate your job schedule

   Once you have found your job, verify that it’s scheduled correctly. Cron schedules are commonly used because they are expressive and powerful, but like regular expressions can sometimes be difficult to read.
   We suggest using Crontab Guru to validate your schedule.

   • Paste the schedule expression from your crontab into the text field on Crontab Guru
   • Verify that the plaintext translation of your schedule is correct, and that the next scheduled execution times match your expectations
   • Check that the effective server timezone matches your expectation. In addition to checking system time using date,
     check your crontab file for TZ or CRON_TZ timezone declarations that would override system settings.
     For example, CRON_TZ=America/New_York

3. Check your permissions

   Invalid permissions can cause your cron jobs to fail in at least 3 ways:

   1. Jobs added as files in a /etc/cron.*/ directory must be owned by root. Files owned by other users will be ignored and you may see a message similar to
      WRONG FILE OWNER in your syslog.
   2. The command must be executable by the user that cron is running your job as. For example if your ubuntu user crontab invokes a script like
      database-backup.sh, ubuntu must have permission to execute the script. The most direct way is to ensure that the ubuntu
      user owns the file and then ensure execute permissions are available using chmod +x database-backup.sh.
   3. The user account must be allowed to use cron. First, if a /etc/cron.allow file exists, the user must be listed. Separately,
      the user cannot be in a /etc/cron.deny list.

   Related to the permissions problem, ensure that if your command string contains a % that it is escaped with a backslash.

4. Check that your cron job is running by finding the attempted execution in syslog

   When cron attempts to run a command, it logs it in syslog.
   By grepping syslog for the name of the command you found in a crontab file you can validate that your job is scheduled correctly and cron is running.

   • Begin by grepping for the command in /var/log/syslog (You will probably need root or sudo access.)

     dev01: ~ $ grep database-backup.sh /var/log/syslog
     Aug  5 4:05:01 dev01 CRON[2128]: (ubuntu) CMD (/var/cronitor/bin/database-backup.sh)

   • If you can’t find your command in the syslog it could be that the log has been rotated or cleared since your job ran. If possible, rule that out by updating the job to run every minute by changing its schedule to * * * * *.
   • If your command doesn’t appear as an entry in syslog within 2 minutes the problem could be with the underlying cron daemon known as crond. Rule this out quickly by verifying that cron is running by looking up its process ID. If cron is not running no process ID will be returned.

     dev01: ~ $ pgrep cron
     323

   • If you’ve located your job in a crontab file but persistently cannot find it referenced in syslog, double check that crond has correctly loaded your crontab file. The easiest way to do this is to force
     a reparse of your crontab by running EDITOR=true crontab -e from your command prompt. If everything is up to date you will see a message like No modification made. Any other
     message indicates that your crontab file was not reloaded after a previous update but has now been updated. This will also ensure that your crontab file is free of syntax errors.

If you can see in syslog that your job was scheduled and attempted to run correctly but still did not produce the expected result you can assume there is a problem with the command you are trying to run.

What to do if your cron job fails unexpectedly

Your cron jobs, like every other part of your system, will eventually fail. This section will walk you through testing your job and spotting common failures.

1. Test run your command like cron does

   When cron runs your command the environment is different from your normal command prompt in subtle but important ways. The first step to troubleshooting is to simulate the cron environment and
   run your command in an interactive shell.

   Run any command like cron does with CronitorCLI

   • Install CronitorCLI for free
   • To force a scheduled cron job to run immediately, use cronitor select to scan your system and present a list of jobs to choose from.
   • To simulate running any command the way cron does, use cronitor shell:
     

     

   Or, manually test run a command like cron does

   • If you are parsing a file in /etc/cron.d/ or /etc/crontab each line is allowed to have an effective «run as» username
     after the schedule and before the command itself. If this applies to your job, or if your job is in another user’s crontab, begin by opening a bash prompt as that user sudo -u username bash
   • By default, cron will run your command using /bin/sh, not the bash or zsh prompt you are familiar with. Double check your crontab file for an optional
     SHELL=/bin/bash declaration. If using the default /bin/sh shell, certain features that work in bash like [[command]] syntax will cause syntax errors under cron.
   • Unlike your interactive shell, cron doesn’t load your bashrc or bash_profile so any environment variables defined there are unavailable in your cron jobs. This is true even if you have a SHELL=/bin/bash declaration. To simulate this, create a command prompt with a clean environment.

     dev01: ~ $ env -i /bin/sh
     $

   • By default, cron will run commands with your home directory as the current working directory. To ensure you are running the command like cron does, run cd ~ at your prompt.
   • Paste the command to run (everything after the schedule or declared username) into the command prompt. If crontab is unable to run your command, this should fail too and will hopefully contain a useful error message.
     Common errors include invalid permissions, command not found, and command line syntax errors.
   • If no useful error message is available, double check any application logs your job is expected to produce, and ensure that you are not redirecting log and error messages.
     In linux, command >> /path/to/file will redirect console log messages to the specified file and command >> /path/to/file 2>&1 will redirect both the console log and error messages.
     Determine if your command has a verbose output or debug log flag that can be added to see additional details at runtime.
   • Ideally, if your job is failing under cron it will fail here too and you will see a useful error message, giving you a chance to modify and debug as needed. Common errors include file not found, misconfigured permissions, and commandline syntax errors.

2. Check for overlapping jobs

   At Cronitor our data shows that runtime durations increase over time for a large percentage of cron jobs. As your dataset and user base grows it’s normal to find yourself in a situation
   where cron starts an instance of your job before the previous one has finished. Depending on the nature of your job this might not be a problem, but several undesired side effects are possible:

   • Unexpected server or database load could impact other users
   • Locking of shared resources could cause deadlocks and prevent your jobs from ever completing successfully
   • Creation of an unanticipated race condition that might result in records being processed multiple times, amplifying server load and possibly impacting customers

   To verify if any instances of a job are running on your server presently, grep your process list. In this example, 3 job invocations are running simultaneously:

   dev01: ~ $ ps aux | grep database-backup.sh
   ubuntu           1343   0.0  0.1  2585948  12004   ??  S    31Jul18   1:04.15 /var/cronitor/bin/database-backup.sh
   ubuntu           3659   0.0  0.1  2544664    952   ??  S     1Aug18   0:34.35 /var/cronitor/bin/database-backup.sh
   ubuntu           7309   0.0  0.1  2544664   8012   ??  S     2Aug18   0:18.01 /var/cronitor/bin/database-backup.sh

   To quickly recover from this, first kill the overlapping jobs and then watch closely when your command is next scheduled to run. It’s possible that a one time failure cascaded into several overlapping instances.
   If it becomes clear that the job often takes longer than the interval between job invocations you may need to take additional steps, e.g.:

   • Increase the duration between invocations of your job. For example if your job runs every minute now, consider running it every other minute.
   • Use a tool like flock to ensure that only a single instance of your command is running at any given time. Using flock is easy. After installing
     from apt-get or yum you only need to prefix the command in your crontab:

   dev01: ~ $ crontab -l
   # Edit this file to introduce tasks to be run by cron.
   # m h  dom mon dow   command
   * * * * *      flock -w 0 /var/cronitor/bin/database-backup.sh

   Read more about flock from the man page.

What to do if nothing else works

Here are a few things you can try if you’ve followed this guide and find that your job works flawlessly when run from the cron-like command prompt but fails to complete successful under crontab.

• First get the most basic cron job working with a command like date >> /tmp/cronlog. This command will simply echo the execution time to the log file each time it runs. Schedule this to run every minute and tail the logfile for results.

• If your basic command works, replace it with your command. As a sanity check, verify if it works.

• If your command works by invoking a runtime like python some-command.py perform a few checks to determine that the runtime version and environment is correct. Each language runtime has quirks
  that can cause unexpected behavior under crontab.

  • For python you might find that your web app is using a virtual environment you need to invoke in your crontab.
  • For node a common problem is falling back to a much older version bundled with the distribution.
  • When using php you might run into the issue that custom settings or extensions that work in your web app
    are not working under cron or commandline because a different php.ini is loaded.

  If you are using a runtime to invoke your command double check that it’s pointing to the expected version from within the cron environment.

• If nothing else works, restart the cron daemon and hope it helps, stranger things have happened. The way to do this varies from distro to distro so it’s best to ask google.

Cronitor automatically finds and monitors cron jobs.  Complete your setup in minutes.

Get Started Free

Anyone who has used cron to schedule tasks in Linux knows that diagnosing and solving errors can be quite difficult. Cron jobs can fail for many reasons, and cron is not known for its error-handling prowess.

This article aims to help with some of the difficulties faced when diagnosing cron errors. We’ll walk through the most common reasons why your cron job isn’t running: schedule errors, environmental changes, depleted resources, and overlapping jobs. We’ll also share some additional troubleshooting tips along with code samples to help get your cron jobs back up and running.

If you’re looking for an alternative to cron for scheduled jobs, we introduce Airplane at the end. Airplane is a developer platform for quickly building internal tools that supports serverless, maintenance-free schedules.

Four reasons your cron job isn’t running

1. Schedule errors

If your cron job isn’t operating as expected, first examine the job’s schedule expressions.

Writing schedule expressions can be tricky. For example, the regular cron expression contains:

<minute> <hour> <day-of-month> <month> <day-of-week> <command>

A single character can make a big difference in an expression’s functionality. For example, replacing - with / in the expression below drastically alters how it works.

Expression 1 : 0-10 14 * * ?

Expression 2 : 0/10 14 * * ?

Expression 1 runs the cron job every minute, starting at 2:00 PM and continuing until 2:10 PM every day. Expression 2 runs the cron job every 10 minutes every day, starting at 2:00 PM and finishing at 2:50 PM.

To address potential scheduling errors, you’ll want to first locate your scheduled job, then make any necessary changes.

Find your job

You can find a job by typing crontab -l to display the current user’s crontab.

Adding a crontab file to /etc/cron.d/ or /etc/crontab is a standard way to generate jobs so be sure to check these locations.

If you don’t see the cron job you’re looking for, another user probably created it. You can run crontab -u username -l to help confirm.

Check permissions

If you still can’t find the job, you should check your permissions. Another user might have created the cron job and you may not be authorized to execute it. Note that the root must own jobs added as files in a /etc/cron.*/ directory. That means that if you don’t have root permissions, you might not be able to access the job.

2. Environmental incompatibilities

You might discover that your job works when run via the command line, but won’t run via cron.

This may be due to the location you’re executing from or the PATH you’re using. The use of relative paths is a common cause of cron issues. In cron, the PATH is set by default to /bin:/usr/bin. Furthermore, only /usr/bin or /bin will be searched for executables.

By default cron jobs are executed from the user’s home directory. This means that if you use relative paths like ../documents/script.sh, cron won’t know where to look for the file.

If you really want to use a file from a location other than your home directory, you should use an absolute path. Absolute paths include the full file path. For example:

1 * * * * /path/to/script.sh

Alternatively, you can define your own PATH and use it in the schedule command as shown below.

It’s also possible that your job includes qualities that are incompatible with cron, such as environment variables and advanced bash features. Errors can happen when using environment variables since cron doesn’t load .bashrc and related files. Similarly, sophisticated bash capabilities can cause difficulties because by default, cron uses /bin/sh.

3. Depleted resources

Cron tasks may fail due to resource depletion. These tasks may run out of disk space or a lack of available space may cause an operating system to be unable to start new threads.

There are various ways to check how much disk space is used and what it’s used by.

For example, you can use the df command. df stands for disk-free and shows the available and used disk spaces in Linux. Here are some ways you can use the df command:

• df -h — human-readable format of available disk space
• df -a — entire disk space used including the available space as 0
• df -T — block filesystem type and disk use, for example, xfs, ext2, ext3, and others

You can also use the du command to view the disk use in kilobytes. Here are some ways you can use the du command:

• du -h — human-readable format of all directories and subdirectories
• du -a — disk use for all files
• du -s — disk space by a specific directory

When you need to see disk size along with disk partitioning information, you can also use fdisk -l to show disk size.

Most of the above commands, particularly the df command, only display disk space for mounted filesystems. You might have multiple operating systems sharing the same disk space or have multiple disks.

In this case, you must first mount the discs before determining the used space. To do so, you can use sudo mount /dev/sdb1 /mnt.

The most common file systems you’ll encounter while checking for disk use are tmpfs, udev, and /dev/loop.

• tmpfs (temporary files) include virtual memory-based temporary files.
• udev stores information on plugged-in devices like USB, network cards, CD ROMs, and external keyboards.
• /dev/loops (or loop devices) are virtual devices to access regular files like block devices. You don’t need to count their space use individually because they’re beneath the root.

If you prefer a GUI over the command line, you can get a graphical view using gnome-disk-utility in GNOME desktops.

4. Overlapping jobs

When a second task begins to run while the first is still running, they overlap. This overlap might cause your cron job to slow down or even crash.

Common measures to take include using a locking mechanism, wrapper scripts, or a concurrency policy to prevent multiple copies of the same cron job from running. When implementing this method, it’s good to save past run times. This way you can identify, for example, if a job that typically takes 30 seconds took more than 2 minutes to finish or if the system is slowing down.

Say you want to initiate a system backup every 30 minutes using a script called script1.sh. For this backup to occur automatically every 30 minutes, you can schedule cron like so:

*/30 * * * * /tmp/script1.sh >> /tmp/script1.log

This saves script1.sh results in script1.log files, making it easy for you to follow — and easy to troubleshoot if the cron job fails.

This is great in theory, but what if the backup requires more than 30 minutes to run? This could result in two backup creations at the same time, which not only slows the system but also includes bulky, repetitive data.

To prevent this, you can use pgrep to check if your script is running and determine the execution time. Using pgrep --list-full bash| grep '/tmp/script1.sh', you can detect every script invocation related to script1.sh, including the present one. If you don’t want to see the current script invocation, you can use grep -v "$$ " to remove any lines beginning with the current PID and the current invocation:

pgrep -a bash | grep -v "^$$ " | grep --quiet '/tmp/script1.sh'

In this code,--quiet is what prevents the output from showing up in the log files.

Then, using the code below, you can see the execution time and halt the job if it overlaps with the previous job:

The output should look something like this:

Checking to see if/when jobs are overlapping can help you determine if you should increase the time between the execution of the two cron jobs.

Other troubleshooting tips

The four examples above are some of most common reasons your cron job may not run as expected.  While these are the most common, there are other possible reasons for cron failure. We outline a couple of other ways to continue to troubleshoot cron below.

Check system logs

When cron tries to execute a command, it records the result in syslog. To check your logs for errors, you’ll need root or sudo access to search for it at /var/log/syslog using this command:

grep filename.sh /var/log/syslog

If you can’t locate your command in the syslog using grep, someone may have deleted the log or moved it to another place.

If a command doesn’t show in syslog after two minutes, the issue might be with the underlying cron daemon, crond.

To help locate the command, you can search for cron-related content directly in the Syslog using the following:

grep CRON /var/log/syslog

Check system resources

Sometimes depleted system resources can cause an error. Suppose your cron job requires you to send an email notification after completing every job. This could fill the available space quickly and slow the system at each iteration.

To help prevent that from happening, you can check if enough system resources are available to run the job you have set up and make adjustments accordingly such as make more memory available by adding more RAM.

Check for infrastructure changes

It’s easy to overlook the cron jobs on each server while making changes to your app configuration and code. These changes can cause cron jobs to fail. This is why it’s worth checking if an infrastructure change may be causing the failure, especially if infra changes were made recently.

These changes could include server updates or migrations, or upgrading or changing routers/devices. These changes could also include things like software updates, internet regulation changes, and firewall or security changes. For example, if a cron job relies on a file path that changes due to data migration, the job will fail. Similarly, a job can fail if it relies on software functionality that is altered or removed with the release of a new version. New security measures might change the level of access required to successfully run a job.

Consider transaction size

Large data sets might cause queries to timeout or file transfers to become excessively slow, resulting in cron job failure.

Although there is no universal transaction size above which your cron jobs are likely to fail, you should check your data set’s size to determine if that might be the cause of your issue. The exact upper limit for transactions handled by your cron job will vary based on your choice of database and architecture, the specific workflow you have in place, and the capability of your system. However, you can typically consider data transactions involving more than 100,000 records to be too large.

A common indication of an overly large dataset is an “out of memory” error, which looks something like this:

Fatal error: Allowed memory size of 262349807 bytes exhausted (tried to allocate 4723456 bytes)...

It’s common to see this error when a job receiving a large response to a query is scheduled to run concurrently with smaller, more frequent jobs. For example, this error might occur if you’re working on a dataset of 200,000 users and try to pull the info for the most recent purchases, which could be 10,000 purchases, simultaneously.

It’s common to handle large queries and other data transactions by breaking the job into smaller fragments. You may be able to use temp tables, offsets, or more complex queries to process the dataset in chunks. If you are able to handle the added overhead, you can also use stored procedures to return subsets of the results you need.

However, although these options create more scalable solutions, they may not always solve query timeouts or slowdowns without creating additional management challenges. Smaller jobs running in parallel can still strain your available memory, so you may need to serialize the workload by rescheduling your jobs to work around the allotted time intervals between them.

Test for code bugs

Sometimes, the failure may not be cron-related. It’s challenging to test cron tasks thoroughly in a development environment, and an issue might only exist in production code. Another troubleshooting tip is to check your code for relevant bugs. For example, maybe you made a recent change to the code base which fails to work as expected or there are other production-related code issues that would also cause issues with your jobs.

Airplane for job scheduling

While cron is one of the most popular methods for scheduling jobs, it may sometimes be difficult to maintain and challenging to troubleshoot. We’ve explored how to diagnose and handle some of the top reasons why cron may not be running, but even experienced developers may find tracking down and solving cron issues time-consuming.

Airplane is a developer platform that lets you turn code into internal apps quickly, supporting quick and easy job scheduling. Using Airplane, you can turn APIs, SQL queries, and scripts into tasks and run those tasks manually or configure them to run on schedules seamlessly. Airplane also comes with permissions, audit logs, and approval flows so you can manage your operations safely.

Check out the Airplane blog for more cron how-to guides such as: How to create Golang cron jobs, How to run cron in containers, and How to start, stop, and restart cron jobs. If you think Airplane could be useful, say hi at hello@airplane.dev or sign up for a free account.

How to fix all of your crontab related woes/problems (Linux)

This is a community wiki, if you notice anything incorrect with this answer or have additional information then please edit it.

First, basic terminology:

• cron(8) is the daemon that executes scheduled commands.
• crontab(1) is the program used to modify user crontab(5) files.
• crontab(5) is a per user file that contains instructions for cron(8).

Next, education about cron:

Every user on a system may have their own crontab file. The location of the root and user crontab files are system dependant but they are generally below /var/spool/cron.

There is a system-wide /etc/crontab file, the /etc/cron.d directory may contain crontab fragments which are also read and actioned by cron. Some Linux distributions (eg, Red Hat) also have /etc/cron.{hourly,daily,weekly,monthly} which are directories, scripts inside which will be executed every hour/day/week/month, with root privilege.

root can always use the crontab command; regular users may or may not be granted access. When you edit the crontab file with the command crontab -e and save it, crond checks it for basic validity but does not guarantee your crontab file is correctly formed. There is a file called cron.deny which will specify which users cannot use cron. The cron.deny file location is system dependent and can be deleted which will allow all users to use cron.

If the computer is not powered on or crond daemon is not running, and the date/time for a command to run has passed, crond will not catchup and run past queries.

crontab particulars, how to formulate a command:

A crontab command is represented by a single line. You cannot use to extend a command over multiple lines. The hash (#) sign represents a comment which means anything on that line is ignored by cron. Leading whitespace and blank lines are ignored.

Be VERY careful when using the percent (%) sign in your command. Unless they are escaped % they are converted into newlines and everything after the first non-escaped % is passed to your command on stdin.

There are two formats for crontab files:

• User crontabs

   # Example of job definition:
   # .---------------- minute (0 - 59)
   # |  .------------- hour (0 - 23)
   # |  |  .---------- day of month (1 - 31)
   # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
   # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)
   # |  |  |  |  |
   # *  *  *  *  *   command to be executed
  

• System wide /etc/crontab and /etc/cron.d fragments

   # Example of job definition:
   # .---------------- minute (0 - 59)
   # |  .------------- hour (0 - 23)
   # |  |  .---------- day of month (1 - 31)
   # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
   # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)
   # |  |  |  |  |
   # *  *  *  *  * user-name  command to be executed
  

Notice that the latter requires a user-name. The command will be run as the named user.

The first 5 fields of the line represent the time(s) when the command should be run.
You can use numbers or where applicable day/month names in the time specification.

• The fields are separated by spaces or tabs.
• A comma (,) is used to specify a list e.g 1,4,6,8 which means run at 1,4,6,8.
• Ranges are specified with a dash (-) and may be combined with lists e.g. 1-3,9-12 which means between 1 and 3 then between 9 and 12.
• The / character can be used to introduce a step e.g. 2/5 which means starting at 2 then every 5 (2,7,12,17,22…). They do not wrap past the end.
• An asterisk (*) in a field signifies the entire range for that field (e.g. 0-59 for the minute field).
• Ranges and steps can be combined e.g. */2 signifies starting at the minimum for the relevant field then every 2 e.g. 0 for minutes( 0,2…58), 1 for months (1,3 … 11) etc.

Debugging cron commands

Check the mail!

By default cron will mail any output from the command to the user it is running the command as. If there is no output there will be no mail. If you want cron to send mail to a different account then you can set the MAILTO environment variable in the crontab file e.g.

MAILTO=user@somehost.tld
1 2 * * * /path/to/your/command

Capture the output yourself

You can redirect stdout and stderr to a file. The exact syntax for capturing output may vary depending on what shell cron is using. Here are two examples which save all output to a file at /tmp/mycommand.log:

1 2 * * * /path/to/your/command &>/tmp/mycommand.log
1 2 * * * /path/to/your/command >/tmp/mycommand.log 2>&1

Look at the logs

Cron logs its actions via syslog, which (depending on your setup) often go to /var/log/cron or /var/log/syslog.

If required you can filter the cron statements with e.g.

grep CRON /var/log/syslog 

Now that we’ve gone over the basics of cron, where the files are and how to use them let’s look at some common problems.

Check that cron is running

If cron isn’t running then your commands won’t be scheduled …

ps -ef | grep cron | grep -v grep

should get you something like

root    1224   1  0 Nov16 ?    00:00:03 cron

or

root    2018   1  0 Nov14 ?    00:00:06 crond

If not restart it

/sbin/service cron start

or

/sbin/service crond start

There may be other methods; use what your distro provides.

cron runs your command in a restricted environment.

What environment variables are available is likely to be very limited. Typically, you’ll only get a few variables defined, such as $LOGNAME, $HOME, and $PATH.

Of particular note is the PATH is restricted to /bin:/usr/bin. The vast majority of «my cron script doesn’t work» problems are caused by this restrictive path. If your command is in a different location you can solve this in a couple of ways:

1. Provide the full path to your command.

    1 2 * * * /path/to/your/command
   

2. Provide a suitable PATH in the crontab file

    PATH=/bin:/usr/bin:/path/to/something/else
    1 2 * * * command 
   

If your command requires other environment variables you can define them in the crontab file too.

cron runs your command with cwd == $HOME

Regardless of where the program you execute resides on the filesystem, the current working directory of the program when cron runs it will be the user’s home directory. If you access files in your program, you’ll need to take this into account if you use relative paths, or (preferably) just use fully-qualified paths everywhere, and save everyone a whole lot of confusion.

The last command in my crontab doesn’t run

Cron generally requires that commands are terminated with a new line. Edit your crontab; go to the end of the line which contains the last command and insert a new line (press enter).

Check the crontab format

You can’t use a user crontab formatted crontab for /etc/crontab or the fragments in /etc/cron.d and vice versa. A user formatted crontab does not include a username in the 6th position of a row, while a system formatted crontab includes the username and runs the command as that user.

I put a file in /etc/cron.{hourly,daily,weekly,monthly} and it doesn’t run

• Check that the filename doesn’t have an extension see run-parts
• Ensure the file has execute permissions.
• Tell the system what to use when executing your script (eg. put #!/bin/sh at top)

Cron date related bugs

If your date is recently changed by a user or system update, timezone or other, then crontab will start behaving erratically and exhibit bizarre bugs, sometimes working, sometimes not. This is crontab’s attempt to try to «do what you want» when the time changes out from underneath it. The «minute» field will become ineffective after the hour is changed. In this scenario, only asterisks would be accepted. Restart cron and try it again without connecting to the internet (so the date doesn’t have a chance to reset to one of the time servers).

Percent signs, again

To emphasise the advice about percent signs, here’s an example of what cron does with them:

# cron entry
* * * * * cat >$HOME/cron.out%foo%bar%baz

will create the ~/cron.out file containing the 3 lines

foo
bar
baz

This is particularly intrusive when using the date command. Be sure to escape the percent signs

* * * * * /path/to/command --day "$(date "+%Y%m%d")"

How to use sudo in cron jobs

when running as a non-root user,

crontab -e

will open the user’s crontab, while

sudo crontab -e

will open the root user’s crontab. It’s not recommended to run sudo commands in a cron job, so if you’re trying to run a sudo command in a user’s cron, try moving that command to root’s cron and remove sudo from the command.

As others have pointed out, cron will email you the output of any program it runs (if there is any output). So, if you don’t get any output, there are basically three possibilities:

1. crond could not even start a shell for running the program or sending email
2. crond had troubles mailing the output, or the mail was lost.
3. the program did not produce any output (including error messages)

Case 1. is very unlikely, but something should have been written in the cron logs. Cron has an own reserved syslog facility, so you should have a look into /etc/syslog.conf (or the equivalent file in your distro) to see where messages of facility cron are sent. Popular destinations include /var/log/cron, /var/log/messages and /var/log/syslog.

In case 2., you should inspect the mailer daemon logs: messages from the Cron daemon usually appear as from root@yourhost. You can use a MAILTO=... line in the crontab file to have cron send email to a specific address, which should make it easier to grep the mailer daemon logs. For instance:

MAILTO=my.offsite.email@example.org
00 15 * * *  echo "Just testing if crond sends email"

In case 3., you can test if the program was actually run by appending another command whose effect you can easily check: for instance,

00 15 * * * /a/command; touch /tmp/a_command_has_run

so you can check if crond has actually run something by looking at
the mtime of /tmp/a_command_has_run.

1. 25.08.2016, 23:16

   
   

   #1

   

   
   Junior Member
   

   

   ---

   Cron Daemon ошибка
   

   Приходит каждый день на почту это ошибка.

   Cron <root@appo> test -x /usr/sbin/anacron || ( cd / && run-parts —report /etc/cron.daily )

   run-parts: /etc/cron.daily/phpsess_cron exited with return code 1

   Подскажите пожалуйста как можно исправить ?

   Ubuntu 16.04.1 LTS (GNU/Linux 4.6.5-x86_64)

   Версия панели : 5.70.0-2016.08.22_11:54

   Последний раз редактировалось appo; 25.08.2016 в 23:23.

   ---

2. 25.08.2016, 23:31

   
   

   #2

   

   
   Senior Member
   

   

   ---

   
   

   Здравствуйте.
   Просмотрите скрипт /usr/local/mgr5/sbin/phpsess_clean.sh, проверьте, все ли пути, указанные в нем, существуют у вас на сервере.
   Запустите вручную, посмотрите вывод.

   ---

3. 26.08.2016, 22:59

   
   

   #3

   

   
   Junior Member
   

   

   ---

   
   

   Сообщение от Dasha
   

   Здравствуйте.
   Просмотрите скрипт /usr/local/mgr5/sbin/phpsess_clean.sh, проверьте, все ли пути, указанные в нем, существуют у вас на сервере.
   Запустите вручную, посмотрите вывод.

   Да вы были правы пути не правильные

   Неправильно

   Код:

   case ${OSTYPE} in
   	DEBIAN)
   		proc_names="apache2"
   		test -f /etc/php5/apache2/php.ini || exit 1
   		gc_maxlifetime=$(php -c /etc/php5/apache2/php.ini -d "error_reporting='~E_ALL'" -r 'echo ini_get("session.gc_maxlifetime");')
   	;;
   	REDHAT)
   		proc_names="httpd httpd.itk"
   		test -f /etc/php.ini || exit 1
   		gc_maxlifetime=$(php -c /etc/php.ini -d "error_reporting='~E_ALL'" -r 'echo ini_get("session.gc_maxlifetime");')
   	;;
   esac

   Правильно

   Код:

   case ${OSTYPE} in
   	DEBIAN)
   		proc_names="apache2"
   		test -f /etc/php/7.0/apache2/php.ini || exit 1
   		gc_maxlifetime=$(php -c /etc/php/7.0/apache2/php.ini -d "error_reporting='~E_ALL'" -r 'echo ini_get("session.gc_maxlifetime");')
   	;;
   	REDHAT)
   		proc_names="httpd httpd.itk"
   		test -f /etc/php.ini || exit 1
   		gc_maxlifetime=$(php -c /etc/php.ini -d "error_reporting='~E_ALL'" -r 'echo ini_get("session.gc_maxlifetime");')
   	;;
   esac

   ---

4. 27.08.2016, 07:49

   
   

   #4

   

   
   Junior Member
   

   

   ---

   
   

   Мне кажется или скрипт не работает потому что в Ubuntu $OSTYPE определяется как linux-gnu
   Проверьте пожалуйста

   ---

5. 27.08.2016, 16:22

   
   

   #5

   

   
   ISPsystem team
   

   

   ---

   
   

   OSTYPE определяется верно:

   root@asas:~# sh -x /usr/local/mgr5/sbin/phpsess_clean.sh
   + [ = -T ]
   + . /usr/local/mgr5/lib/pkgsh/ispmgr_pkg_funcs.sh
   + MGRDIR=/usr/local/mgr5
   + . /usr/local/mgr5/lib/pkgsh/core_pkg_funcs.sh
   + MGRDIR=/usr/local/mgr5
   + MGRCTL=/usr/local/mgr5/sbin/mgrctl
   + OSType
   + [ -f /etc/redhat-release ]
   + [ -f /etc/debian_version ]
   + echo DEBIAN
   + return
   + OSTYPE=DEBIAN
   + OSType
   + [ -f /etc/redhat-release ]
   + [ -f /etc/debian_version ]
   + echo DEBIAN
   + return
   + [ DEBIAN = DEBIAN ]
   + MYSQL_SERVICE=mysql
   + TMPCNF=/usr/local/mgr5/tmp/.2022.cnf
   + IsMgrExist ispmgrnode
   + [ #ispmgrnode = # ]
   + grep -qE ^managers+ispmgrnode$ /usr/local/mgr5/etc/mgrlist.conf
   + return 1
   + ISPMGR=ispmgr
   + MGRDIR=/usr/local/mgr5
   + ISPMGRCTL=/usr/local/mgr5/sbin/mgrctl -m ispmgr
   + proc_names=apache2
   + test -f /etc/php5/apache2/php.ini
   + exit 1

   root@asas:~# cat /etc/issue
   Ubuntu 16.04.1 LTS n l

   Выполните у себя на сервер команду sh -x /usr/local/mgr5/sbin/phpsess_clean.sh и покажите результат

   ---

6. 27.08.2016, 22:57

   
   

   #6

   

   
   Junior Member
   

   

   ---

   
   

   root@appo:~# sh -x /usr/local/mgr5/sbin/phpsess_clean.sh
   + [ = -T ]
   + . /usr/local/mgr5/lib/pkgsh/ispmgr_pkg_funcs.sh
   + MGRDIR=/usr/local/mgr5
   + . /usr/local/mgr5/lib/pkgsh/core_pkg_funcs.sh
   + MGRDIR=/usr/local/mgr5
   + MGRCTL=/usr/local/mgr5/sbin/mgrctl
   + OSType
   + [ -f /etc/redhat-release ]
   + [ -f /etc/debian_version ]
   + echo DEBIAN
   + return
   + OSTYPE=DEBIAN
   + OSType
   + [ -f /etc/redhat-release ]
   + [ -f /etc/debian_version ]
   + echo DEBIAN
   + return
   + [ DEBIAN = DEBIAN ]
   + MYSQL_SERVICE=mysql
   + TMPCNF=/usr/local/mgr5/tmp/.31911.cnf
   + IsMgrExist ispmgrnode
   + [ #ispmgrnode = # ]
   + grep -qE ^managers+ispmgrnode$ /usr/local/mgr5/etc/mgrlist.conf
   + return 1
   + ISPMGR=ispmgr
   + MGRDIR=/usr/local/mgr5
   + ISPMGRCTL=/usr/local/mgr5/sbin/mgrctl -m ispmgr
   + proc_names=apache2
   + test -f /etc/php/7.0/apache2/php.ini
   + php -c /etc/php/7.0/apache2/php.ini -d error_reporting=’~E_ALL’ -r echo ini_get(«session.gc_maxlifetime»);
   + gc_maxlifetime=1440
   + awk -F= {print $2}
   + /usr/local/mgr5/sbin/mgrctl -m ispmgr pathlist elid=apache-vhosts
   + vhost_dir=/etc/apache2/vhosts
   + test -n /etc/apache2/vhosts
   + [ ! -d /etc/apache2/vhosts/appo ]
   + + sort+
   uniq
   sed s/»//g
   + awk {print $3}
   + grep -Rh php_admin_value session.save_path /etc/apache2/vhosts/appo/appo.www
   + save_path_dirs=/var/www/appo/data/mod-tmp
   /var/www/appo/data/mod-tmp
   + [ -d /var/www/appo/data/mod-tmp ]
   + pidof apache2
   + find /proc/15539/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/14748/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/14746/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/14745/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/14741/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/14740/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find /proc/4655/fd -ignore_readdir_race -lname /var/www/appo/data/mod-tmp/sess_* -exec touch -c {} ;
   + find -O3 /var/www/appo/data/mod-tmp -ignore_readdir_race -depth -mindepth 1 -name sess_* -type f -cmin +1440 -delete
   ][ -d /var/www/appo/data/mod-tmp

   да все ок

   ---

7. 29.09.2016, 00:34

   
   

   #7

   

   
   Junior Member
   

   

   ---

   
   

   Можно узнать когда будет обновление в самом ISPmanager потому что мне приходится после каждого обновления менять все заново в ручную

   ---

8. 29.09.2016, 03:23

   
   

   #8

   
   

   Как временный костыль снимите с файла права на запись
   Панель не сможет его изменить при обновлении

   ---

9. 29.09.2016, 11:57

   
   

   #9

   

   
   ISPsystem team
   

   

   ---

   
   

   Можно узнать когда будет обновление в самом ISPmanager потому что мне приходится после каждого обновления менять все заново в ручную

   Очередное обновление панели ожидается 04.10.2016.

   Когда будет обновление, где исправлена данная ошибка, пока, к сожалению, не известно.

   ---

10. 29.09.2016, 21:36

    
    

    #10

    

    
    Junior Member
    

    

    ---

    
    

    Сообщение от Mobiaaa
    

    Как временный костыль снимите с файла права на запись
    Панель не сможет его изменить при обновлении

    Я сделал по другому костыль, добавив питон скрипт в cron.daily который проверяет строки каждый день думаю временно пойдет если есть варианты получше предлагайте.

    Код:

    #!/bin/python
    
    # Read in the file
    filedata = None
    with open('/usr/local/mgr5/sbin/phpsess_clean.sh', 'r') as file :
      filedata = file.read()
    
    # Replace the target string
    filedata = filedata.replace('/etc/php5/apache2/php.ini', '/etc/php/7.0/apache2/php.ini')
    
    # Write the file out again
    with open('/usr/local/mgr5/sbin/phpsess_clean.sh', 'w') as file:
      file.write(filedata)

    ---

Cron daemon sending email continuously is a bug that is sometimes seen after a cPanel update.

Here at Bobcares, we resolve web hosting technical issues such as this as part of our Server Management services for Web Hosts, Digital Marketing Agency, and other online service providers.

Today we’ll take a look at what causes this cron daemon error and how to fix it.

Why does cron daemon sending email continuously?

Let’s first see why does the cron daemon send emails continuously.

Many of our customers use cPanel because of its user-friendly interface. However, errors are not free from this popular control panel.

One such error is cron daemon sending emails continuously every 10 minutes. Here, the mail normally contains some weird cronjob error with the subject having a cron command. The sender will be root@hostname. Also, the mail body will contain cPanel exception error. As a result, the mailbox will be full of such spam emails.

We’ve seen in most of the cases this issue occurs if auto-update is on. Therefore, we always take a close look at the error message in the mail and find the root cause. The error can be due to several reasons. One of the common reasons is an incomplete update.

How we fix cron daemon sending email issues?

Mostly, shared hostings have cPanel and some of our clients with VPS hosting are having it as well. As a result, we fix errors like this very often.

Now let’s take a look at a recent request that we received from our customer. Here is the error message.

According to him, all of a sudden such emails kept coming every 10 minutes. He suspects that some auto-update is causing it as he didn’t touch anything.

Now let’s see how our Support Engineers fix this error.

We were able to recreate the error and found that “cpanel-dovecot-solr” service was down and it was failing to restart.

Also, we found that there were still some pending updates for cPanel. So updating it manually should fix the issue.

We ran the below cPanel’s script from the command line to install available updates.

/scripts/upcp --force

As a result, it updated successfully and restarting the pending services stopped further error mails.

[Having trouble in fixing Cpanel errors? – We’ll fix it for you.]

Conclusion

In short, the error occurs due to broken updates. Today, we saw how our Support Engineers fix errors related to Cpanel.

var google_conversion_label = «owonCMyG5nEQ0aD71QM»;