calo/calo_cdb/README.org

0001 #+title: README
0002
0003 * Overview
0004
0005 This package automates the generation of the CDB maps for the calorimeters. Specifically, the following maps are considered:
0006 - BadTowerMap: CEMC
0007 - meanTime: CEMC, HCALIN and HCALOUT
0008 - fracBadChi2: CEMC, HCALIN and HCALOUT
0009
0010 ** Steps
0011
0012 1) The ~FileCatalog~ is queried for all runnumber, tag, file paths, and time given a run type. By default the run type is set to ~run3auau~.
0013 2) The result is reduced as follows: Only the tag corresponding to the latest time among duplicate instaces of run numbers is kept. For example, if there are two instances (run, tag, time) of (A,B,C) and (A,D,E) where E > C then (A,B,C) is dropped and only (A,D,E) is kept.
0014 3) Using the ~filter-datasets~ binary, the CDB (Calibrations Database) is pinged given the run number and specific calibration name. If no result is returned by the CDB or if the tag in the calibration name does not match the latest tag, then the pair of run number and tag is kept to produce the calibration files. Otherwise the pair is removed since the latest calibration already exists in the CDB. A list of all such runs is written to ~{runtype}-process.csv~.
0015 4) Using the ~{runtype}-process.csv~ file, a unique file is created for each run number which consists of all of the ~HIST_CALOQA~ for that run number.
0016 5) A condor submission directory is created that processes one job per run number and uses the ~genStatus~ binary to combine all of the relevant histograms from the dataset file and write out the CDB files for the specified calibrations.
0017
0018 * Prerequisites
0019 - Python 3.13.0
0020 - Additional Python Packages:
0021   - SQLAlchemy 2.0.41
0022   - pandas     2.2.3
0023
0024 * runProd.py Script Details
0025
0026 To get a quick list of all options allowed, run the script using the ~-h~ flag as follows: ~runProd.py -h~.
0027 Some key flags:
0028 - ~--run-type~: By default is set to ~run3auau~ but can also work with ~run2auau~ or ~run2pp~.
0029 - ~--output~: By default is set to ~test~. This directory is where all relevant log files and condor files are stored.
0030 - ~--condor-log-dir~: By default is set to ~/tmp/<USER>/dump~. This directory is where the condor log files are stored. As long as the location is under ~/tmp~ it should be fine.
0031 - ~--do-condor-submit~: By default this is disabled. Enabling this option submits the condor jobs as needed. Useful for embedding the script in a cron job. Note: Jobs are only submitted if there exists a run with outdated or missing calibrations.
0032 - By default the ~--bin-filter-datasets~ and ~--bin-genStatus~ are set empty as they are determined on on-the-fly through the ~$OFFLINE_MAIN/bin~ path where the binaries for the key routines are stored.
0033
0034 Example:
0035 #+begin_src bash
0036 runProd.py --run-type run3auau --output run3auau --verbose
0037 #+end_src
0038
0039 Example Cron Job (Once a day at midnight):
0040 #+begin_src
0041 0 0 * * * runProd.py --run-type run3auau --output run3auau --verbose --do-condor-submit
0042 #+end_src
0043
0044 * runProd.C Macro Details
0045
0046 ** Overview
0047 This is a macro designed to automate the generation of calorimeter calibration files for the sPHENIX Calibrations Database (CDB). The macro checks if up-to-date calibrations already exist for a given run and dataset. If they are missing or outdated, it invokes the necessary calo_cdb library to process the data and generate the new calibration files.
0048
0049 ** Features
0050 - Modern C++20 Implementation: Utilizes modern C++ features like std::filesystem for robust path handling, std::regex for reliable filename parsing, and std::format for clean output.
0051 - Object-Oriented Design: All logic is encapsulated within a ProductionManager class, ensuring clean state management and eliminating global variables.
0052 - Intelligent CDB Checking: Automatically connects to the CDB using a specified global tag and checks a predefined list of essential calorimeter calibrations (CEMC_BadTowerMap, CEMC_meanTime, etc.).
0053 - Outdated Calibration Detection: Compares the production dataset tag found in the CDB with the one from the input file to determine if existing calibrations are outdated.
0054 - Robust Error Handling: Implements comprehensive argument validation and uses exceptions for error handling to ensure predictable behavior.
0055 - Force Option: Includes a ~--force~ flag to override the CDB check and regenerate calibrations regardless of their current status.
0056   Examples
0057
0058 ** Usage
0059 1. Standard Check and Generate:
0060 Check if run 68006 from the new_newcdbtag_v008 dataset needs new calibrations. If so, generate them in the cdb_output directory.
0061
0062 #+begin_src bash
0063 root -b -l -q 'runProd.C("/path/to/lists/68006_new_newcdbtag_v008.list", "cdb_output")'
0064 #+end_src
0065
0066 2. Force Regeneration:
0067 Regenerate calibrations for run 68006, overwriting any existing entries, using a specific database tag.
0068
0069 #+begin_src bash
0070 root -b -l -q 'runProd.C("/path/to/lists/68006_new_newcdbtag_v008.list", "cdb_output", 1, "newcdbtag")'
0071 #+end_src
0072
0073 ** Workflow Logic
0074
0075 1. Argument Parsing & Validation: The macro ensures the input list exists and the output directory is valid.
0076
0077 2. Run Info Extraction: The run number and dataset tag are parsed directly from the input filename using a regular expression.
0078
0079 3. CDB Status Check:
0080 - If the ~--force~ flag is not used, the macro connects to the CDB.
0081 - It iterates through a hardcoded list of essential calibration names.
0082 - For each name, it checks if an entry exists for the given run number and if the production tag matches the one from the input file.
0083 - If any calibration is missing or has an outdated tag, the check fails.
0084 - If all calibrations are present and up-to-date, the macro prints a message and exits successfully without doing any work.
0085 - Calibration Generation: If the CDB check fails or if ~--force~ is used, the macro instantiates the GenStatus class and calls its ~process()~ method, passing the input list and output directory to generate the new calibration files.