Personal tools
You are here: Home Data and software NuGrid Data server Instructions
Navigation
Log in


Forgot your password?
« July 2017 »
Su Mo Tu We Th Fr Sa
1
2345678
9101112131415
16171819202122
23242526272829
3031
 
Document Actions

Instructions

Instructions for NuGrid members and users

Introduction

The CADC data server is meant for permanent or semi-permanent curated data associated with a NuGrid project. Typically the data would contain a complete run directory, say of a MESA or mppn run, or ppn trajectory runs with output. The typical progression would be an upload to data-team where Nugrid members can look at, check, use data. As the paper about the data is written and published the final version is moved to data.

The CADC VOspace is not meant to be used for quick data exchange of temporary types of data. You can use dropbox for that.


First step: register with CADC at http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/auth/register.html - this will give you a CADC user account.
Second step:  contact the membership POC to be added to the nugrid-team group (see below, access to nugrid-write will be given when needed)


The easiest way to access the CADC server is through the web interface.  For other methods see the VOspace documentation and/or the information below.  If you are moving a large chunk of files, it is recommended that you use the web interface or the python tools (vcp) directly rather than a mount point.



Briefest possible instructions python tools and VO filesystem:

Either watch the instructional video on the NuGrid YouTube channel or follow the instructions below.

  1. for Mac: Install FUSE (http://osxfuse.github.com/); for Linux check VOspace filesystem info
  2. sudo easy_install vos
  3. mount the VOspace
    • in some appropriate place, e.g. your home, create a directory nugrid_tmp
    •  mountvofs --readonly --cache_nodes --log=/path/to/your/dir/nugrid_tmp/vos_log --cache_dir=/path/to/your/dir/nugrid_tmp/vosCache --mountpoint=/path/to/home/nugrid --vospace=vos:nugrid
    • now the public NuGrid data on the NuGrid VOspace data.nugridstars.org should now be mounted on and available in /path/to/home/nugrid
    • if you have nugrid privileges you may use your CADC/nugrid registration for access of the NuGrid internal collaboration data (see below), i.e. use getCert before mounting
  4. go over to http://nugridpy.phys.uvic.ca and easy_install or pip install nugridpy
  5. now, plotting a 2Msun, Z=0.02 HRD diagram (to get you started) would involve ...


import mesa as ms
ms.set_nugrid_path('/path/to/home/nugrid')
m2se=ms.history_data(mass=2, Z=0.01)
m2se.hrd()

and to plot a Ba profile at cycle 45000:

import nugridse as nuse
nuse.set_nugrid_path('/path/to/home/nugrid')
m2=nuse.se(mass=2, Z=0.01)
m2.plot('mass','Ba-138',fname=45000)


Use mountvofs --help to get more options. If you have more than one user mounting the vospace on the same machine (as may happen for example on helix) make sure each user is using their own mountpoint, cache_dir and log file (to be specified with --log).

Moving data to and from the CADC:

The mounted virtual file system is very efficient for browsing datasets or data exploration. Faster visualisation and analysis can also be done using WENDI (wendi.nugridstars.org). However, for downloading or uploading data from/to the CADC it is recommended that you do not cp or rsync from the mounted file system, and instead use the vcp command-line tool that is installed as part of vos (see installation section below). This can be done like so:

vcp -v vos:nugrid/data-team/path/to/some/data /local/path
vcp -v /local/path vos:nugrid/data-team/path


Instructions python tools and VO filesystem:

  1. > sudo easy_install vos
    package correctly, put on http://pypi.python.org , can also be installed in a local directory using > sudo easy_install --install-dir . vos
  2. > getCert
    Gets the certificate for the CADC data server. Use your CADC username and password to log in. The certificate has an expiration date, means, if you cannot mount the file system, run a getCert command again. This might solve the problem.
  3. > vls vos:nugrid
    This command displays information about the data structure on the CADC server. Useful for debugging the mounting command. Note that the ./vls is  necessary if the installation directory of vos is in a local directory and you are in the respective folder (see special option in step 1)
  4. Install FUSE, available on http://osxfuse.github.com/
  5. > mountvofs --readonly --cache_nodes --cache_dir=/tmp/vosCache --mountpoint=/tmp/NuGrid --vospace=vos:nugrid
    Mount the actual folder 'nugrid' (see end of command) on your file system. The folder where it is mounted is /tmp/NuGrid. The cahce_dir is important if you do not want to wait forever to look at data repetitively. The standard cache size is 50 GB on your hard disk, meaning, up to the last 50 GB of the data you looked at from the CADC server is stored locally


About CADC NuGrid VOspace

The data is served by CADC, and our web address http://data.nugridstars.org points to it.

Raw CADC link: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/vosui/#nugrid

CADC instructions: http://canfar.phys.uvic.ca/wiki/index.php/VOStorage



Structure

There are presently two data areas set up:
  1. data - contains published data sets that should be available to our users
  2. data-team - contains data accessible only to NuGrid members

Groups

Christian Ritter has now authority to add and remove people from nugrid-write and the other groups. Please contact him to add you to the nugrid-write group if needed. In order to protect the data integrity (remember we are not protected against accidental deleting) we will add people for particular write tasks and then remove them again.So, please let Christian know (via a message to nugrid-team!) when you are done writing (so that he can remove you again), and say what you have written so that we all know.

We have established 3 groups on the data server:
  1. nugrid - can read data
  2. nugrid-team - can read data and data-team
  3. nugrid-write - can read and write data and data-team
  4. nugrid-admin - can add and remove people (Christian & Falk)

Access

Access can be in (at least) three ways:

  • web interface
  • python tools
  • VOSpace



Instructions python tools and VO filesystem:

  1. > sudo easy_install vos
    package correctly, put on http://pypi.python.org , can also be installed in a local directory using > sudo easy_install --install-dir . vos
  2. > getCert
    Gets the certificate for the CADC data server. Use your CADC username and password to log in. The certificate has an expiration date, means, if you cannot mount the file system, run a getCert command again. This might solve the problem.
  3. > vls vos:nugrid
    This command displays information about the data structure on the CADC server. Useful for debugging the mounting command. Note that the ./vls is  necessary if the installation directory of vos is in a local directory and you are in the respective folder (see special option in step 1)
  4. Install FUSE, available on http://osxfuse.github.com/
  5. > mountvofs --readonly --cache_nodes --cache_dir=/tmp/vosCache --mountpoint=/tmp/NuGrid --vospace=vos:nugrid
    Mount the actual folder 'nugrid' (see end of command) on your file system. The folder where it is mounted is /tmp/NuGrid. The cahce_dir is important if you do not want to wait forever to look at data repetitively. The standard cache size is 50 GB on your hard disk, meaning, up to the last 50 GB of the data you looked at from the CADC server is stored locally


Usually, the folder should stay mounted, even when switching your workplace and being connected with a different IP address. For remounting, get the certificate again (getCert) and then use the mountvofs command again. If you do not want to look it up on the plone everytime, useful is to grep your bash_history file:

> grep mountvofs .bash_history

You can unmount the system via umount on Mac (fsumount on Linux?).


If you do have write access it is recommended to use a separate mount point for write actions, and use that write mount point only for write operations, in order to make sure that your everyday read use of the mount point does not accidentally delete some of our precious data.

For example, I recommend you create two aliases, like this

alias mnt.vos='mountvofs --readonly --cache_nodes --cache_dir=/tmp/vosCache --mountpoint=/tmp/nugrid --vospace=vos:nugrid'
alias mnt.vos_w='mountvofs --cache_nodes --cache_dir=/tmp/vosCache_w --mountpoint=/tmp/nugrid_w --vospace=vos:nugrid'

Also: use the --log option to separate out log files for each user.

alias mnt_vos='mountvofs --log=/rpod2/fherwig/tmp/vos_log_FH --cache_nodes --cache_dir=/rpod2/fherwig/tmp/vosCache_FH --mountpoint=/tmp/NuGrid_FH --vospace=vos:nugrid'

Also, remember that if you have uploaded something it may be necessary to unmount and mount again the VOspace to see the data on the server.

Useful further commands can be found at: https://wiki.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/canfar/index.php/VOSpace_filesystem

FAQ

  • Error: 'RuntimeError: threads can only be started once' when using the python tools
    Ask yourself: did you use Ctrl + C? Restart python session and be more patient. Data needs time to get transferred to your computer.
  • Error: '/NuGrid is itself on a OSXFUSE volume' when mounting the CADC server
    Check if the volume is already mounted
    Renew the certificate (run getCert command)
  • In general: the VOS package seems to upgraded on a regular basis. Don't forget to check and upgrade every not and then. Run:
    easy_install --upgrade vos
  • Problem writing onto the server: 1) Make sure that the vospace is mounted writeable (that you don't use the --readonly option). 2) Unmount the folder, update your certificate, and mount the folder again. 3) Check on data.nugridstar.org (using the online interface) if your data is uploaded and you just cannot see it. 4) Sometimes if data are there and not visible using ls, unmount and remount again.
  • Is the VOspace data backed up? The vospace heavily mirrored so we are protected against catastrophe.  If a user executes a 'delete' commend there is a lag between when a file is marked for deletion (user thinks file is deleted) and when that file is actually deleted.  That lag can be very short (seconds) or quite long  (days) depending on operational situations in the CADC.  If you have a file you are hoping to retrieve from the delete pile, I suggest contacting John Ouellet directly as he is the only one who can stop the physics delete. John Ouellette <John.Ouellette@nrc-cnrc.gc.ca>
  • "No command 'getCert' found, did you mean: Command 'getcert' from package 'certmonger' (universe); getCert: command not found": The install destination for python packages is set by the local python setup not by the vos installer.  I have noticed that many installers are now putting user installed packages into ${HOME}/.local/bin. The solution to this issue is then to add ${HOME}/.local/bin to your PATH in your .bashrc file, i.e. export PATH=${HOME}/.local/bin:$PATH"


Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: