Debugging CernVM-FS Mount Problems

When CernVM-FS refuses to mount a repository, it typically shows like

> ls /cvmfs/sft.cern.ch
ls: /cvmfs/sft.cern.ch: No such file or directory

or like

> cvmfs_config probe
Probing /cvmfs/sft.cern.ch... Failed!

The following few steps can help to identify and resolve the cause of the problems. Note that the automounter caches negative attempts to access a directory. So if the source of the problem is fixed, it might be necessary to run sudo service autofs reload.

  • Run sudo cvmfs_config chksetup. This script detects and reports many common configuration problems.
  • Check the system log for messages from the cvmfs2 process. In many cases, CernVM-FS will report why it fails to mount.
  • If the file system appears to be hanging, it might have been interrupted during a reload operation. Try to run sudo cvmfs_config killall. Note that this will forcefully terminate all currently mounted CernVM-FS repositories and applications using them.
  • Try to mount manually, avoiding the automounter. To mount manually, create an empty directory /mnt/test and run sudo mount -t cvmfs sft.cern.ch /mnt/test
    If a manual mount works but automounter fails to mount, most likely SElinux is interfering.  You can check /var/log/audit/audit.log or temporarly disable SElinux. If it's not SElinux, increase the verbosity of the autmounter (careful, this is a disruptive operation): set LOGGING="debug" in /etc/sysconfig/autofs (RHEL, SuSE) or /etc/default/autofs (Ubuntu). Restart the autofs service (sudo service autofs restart) to make the change effective. Detailed automounter progress will be logged to the system log.
  • Check if the file system hosting the CernVM-FS cache has at least 15% extra space on top of the configured CVMFS_QUOTA_LIMIT.
  • Check if you are affected by any of the known issues.
  • Check if the cache got corrupted using cvmfs_config -j4 fsck.
    Corrupted files in the cache should be very rare. They can be a result of silent disk corruption but if they appear regularly they more likely indicate a bug in CernVM-FS. In this case, please submit a bug report.

Careful: the next steps affect and disrupt all repositories, not just the one being debugged.

 

  • Try with a clean cache. To clean the cache, run cvmfs_config wipecache.
  • Trigger the mount failure in debug mode. To enable the debug mode, set CVMFS_DEBUGLOG=/tmp/cvmfs.log in /etc/cvmfs/default.local. Submit a bug report and attach the resulting debug log in addition to the bugreport tarball. Note that the debug mode quickly produces large debug log files and it is not suitable for production use.

 

You are here