Debugging CernVM-FS Mount Problems
When CernVM-FS refuses to mount a repository, it typically shows like
> ls /cvmfs/sft.cern.ch ls: /cvmfs/sft.cern.ch: No such file or directory
> cvmfs_config probe Probing /cvmfs/sft.cern.ch... Failed!
The following few steps can help to identify and resolve the cause of the problems. Note that the automounter caches negative attempts to access a directory. So if the source of the problem is fixed, it might be necessary to run
sudo service autofs reload.
sudo cvmfs_config chksetup. This script detects and reports many common configuration problems.
- Check the system log for messages from the
cvmfs2process. In many cases, CernVM-FS will report why it fails to mount.
- If the file system appears to be hanging, it might have been interrupted during a reload operation. Try to run
sudo cvmfs_config killall. Note that this will forcefully terminate all currently mounted CernVM-FS repositories and applications using them.
- Try to mount manually, avoiding the automounter. To mount manually, create an empty directory /mnt/test and run
sudo mount -t cvmfs sft.cern.ch /mnt/test
If a manual mount works but automounter fails to mount, most likely SElinux is interfering. You can check /var/log/audit/audit.log or temporarly disable SElinux. If it's not SElinux, increase the verbosity of the autmounter (careful, this is a disruptive operation): set
LOGGING="debug"in /etc/sysconfig/autofs (RHEL, SuSE) or /etc/default/autofs (Ubuntu). Restart the autofs service (
sudo service autofs restart) to make the change effective. Detailed automounter progress will be logged to the system log.
- Check if the file system hosting the CernVM-FS cache has at least 15% extra space on top of the configured
- Check if you are affected by any of the known issues.
- Check if the cache got corrupted using
cvmfs_config -j4 fsck.
Corrupted files in the cache should be very rare. They can be a result of silent disk corruption but if they appear regularly they more likely indicate a bug in CernVM-FS. In this case, please submit a bug report.
Careful: the next steps affect and disrupt all repositories, not just the one being debugged.
- Try with a clean cache. To clean the cache, run
- Trigger the mount failure in debug mode. To enable the debug mode, set
CVMFS_DEBUGLOG=/tmp/cvmfs.login /etc/cvmfs/default.local. Submit a bug report and attach the resulting debug log in addition to the bugreport tarball. Note that the debug mode quickly produces large debug log files and it is not suitable for production use.