Commands to Access Local Data

Contents

• Introduction
• Important technical detail
• Accessing data from DFS
• Accessing_data inside ROOT
  • Open a single file in ROOT
  • Open a multiple file in ROOT
• Accessing_data using gfal2 tools
  • Creating a directory in EOS
  • Delete an empty directory in EOS
  • Copy data from local to EOS
  • Delete data from EOS
  • List EOS data directory
  • Renaming a folder in EOS

Introduction

Starting in 2023 the CMS Tier-2 at Purdue uses EOS as a local distributed filesystem for storing users' and experiment's data.

Important technical detail

The local Distributed Filesystem EOS is intended and optimized for long-term storage of large (ROOT) datafiles, and not for small text files (e.g. source code), nor is it intended for interactive use in the same way as other POSIX-compliant filesystems (e.g. home directories, or Depot) are.

You cannot open a file in EOS with a text editor and change its contents. You cannot use regular UNIX commands like 'cp' or 'rm' or 'mkdir' in EOS.

The read-only FUSE mount on the Front-End machines is intended only to facilitate you in navigating its contents ('ls', 'find') and for occasional reading of ROOT datafiles while developing your code. All large-scale access to EOS during your analysis is supposed to happen via `xrootd` and `gfal` commands as explained below.

XRootD protocol should be the main access method, using the gfal-* and xrd* commands described below.

Accessing data via xrootd

Data in DFS can be read from 'eos.cms.rcac.purdue.edu' with a valid grid certificate.

xrdcp root://eos.cms.rcac.purdue.edu//store/path/to/file.root  /path/to/my.root 

Accessing data inside ROOT

Use the following methods to load files in ROOT.

Open a single file in ROOT

root [0] TFile* f = TFile::Open("root://eos.cms.rcac.purdue.edu//store/group/ewk/DY/May10ReReco/ntuple_skim_9_6_ZgF.root");

Open multiple files in ROOT

Create text file in the directory you are going to start ROOT, call it "xrootdfiles.txt". Populate xrootdfiles.txt with the files you would like to open.

root://eos.cms.rcac.purdue.edu//store/path/to/file1.root
root://eos.cms.rcac.purdue.edu//store/path/to/file2.root
.
.
.
root://eos.cms.rcac.purdue.edu//store/path/to/fileX.root

Add the files to a TFileCollection object in ROOT:

root [0] TFileCollection* c1 = new TFileCollection("data","data");
root [1] c1->AddFromFile("xrootdfiles.txt");
root [2] c1->Print("L");
TFileCollection data - data contains: 3 files with a size of 0 bytes, 
0.0 % staged - default tree name: '(null)'

Accessing data using gfal tools

Data at the Purdue Tier-2 can be accessed from 'eos.cms.rcac.purdue.edu' with a valid grid certificate.

# save the original path into a new environment variable
export LD_LIBRARY_PATH_DEFAULT=$LD_LIBRARY_PATH

# run your work that requires cmsenv
# ...

# reset the LD_LIBRARY_PATH environment variable
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_DEFAULT

Creating a directory in EOS

In the following commands <username> stands for your CERN lxplus ID.

gfal-mkdir davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/<directoryname>

Delete a directory and all its contents in EOS (be careful!)

gfal-rm -r davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/<directoryname>

Copy data from local to EOS

gfal-copy file:////tmp/test.root davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/test.root

Delete data from EOS

gfal-rm davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/test.root

List EOS data directory

gfal-ls davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>

Renaming a folder in EOS

gfal-rename davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/folder1 davs://eos.cms.rcac.purdue.edu:9000/store/user/<username>/folder2