Customization and Global Configuration

The config.yml file

This package uses a config.yml configuration file to customize certain properties. You can control a number of key features such as where your data will download to. SWxSOC packages look for this configuration file in a platform-specific directory, which you can see the path for by running:

>>> import swxsoc
>>> swxsoc.print_config()  

Using your own config.yml file

To maintain your own customizations, you must place your customized config.yml inside the appropriate configuration folder (which is based on the operating system you are working on). The AppDirs module provided by the sunpy package is used to determine where to look for your configuration file.

Warning

Do not edit the config.yml file directly in the Python package as it will get overwritten every time you re-install or update the package.

You can copy the file below, customize it, and then place your customized config.yml file inside your config folder.

If you work in our developer environment you can place your configuration file in this directory:

/home/vscode/.config/swxsoc/

You can also specify the configuration directory by setting the environment variable SWXSOC_CONFIGDIR to the path of your configuration directory. For example, you can set the environment variable in your terminal by running:

export SWXSOC_CONFIGDIR=/path/to/your/config/dir

If you do not use our developer environment, you can run the following code to see where to place it on your specific machine as well:

>>> from swxsoc import util
>>> print(util.config._get_user_configdir())
/home/vscode/.config/swxsoc

Note

For more information on where to place your configuration file depending on your operating system, you can refer to the AppDirs module docstrings.

Customizing the Mission Configuration

The configuration file supports keeping multiple mission configurations. You must select one mission to be used either directly in the configuration file or via an environmental variable.

In the config.yml file, you can specify the mission by setting the selected_mission variable. Here is an example snippet from a config.yml file:

selected_mission: "mission_name"
missions_data:
  mission_name:
    file_extension: ".txt"
    instruments:
      - name: "Instrument1"
        shortname: "Inst1"
        fullname: "Instrument 1"
        targetname: "Target 1"
      - name: "Instrument2"
        shortname: "Inst2"
        fullname: "Instrument 2"
        targetname: "Target 2"

You can override the selected mission by setting the SWXSOC_MISSION environment variable. This is useful for scenarios such as running in different environments (e.g., Lambda containers). For example:

export SWXSOC_MISSION=another_mission

Reconfiguring for Testing

For testing purposes, you might need to reload the configuration after making changes to the config.yml file. You can use the _reconfigure function to reload the configuration during your testing process. This function reloads the configuration and updates the global config variable.

from swxsoc import _reconfigure

# Make changes to the config.yml file
# ...

# Reconfigure the module to reload the configuration
_reconfigure()

To learn more about how to set up your development environment, see Developer Environment.

See below (A sample config.yml file) for an example configuration file.

A sample config.yml file

(download)

# Configuration
# This is the default configuration file

general:
  # Time Format to be used for displaying time in output (e.g. graphs)
  # The default time format is based on ISO8601 (replacing the T with space)
  # note that the extra '%'s are escape characters
  time_format: "%Y-%m-%d %H:%M:%S"

  # Default Key to use for Time Series time data in CDF Files
  # Additional Time keys can be used, however the following is treated
  # as the default and primary time variable key
  default_timeseries_key: Epoch

selected_mission: demo

missions_data:
  demo: # A generic mission template for demonstration
    file_extension: cdf # The File Extension Used for Archive Files and Science File Formats
    min_valid_time: 2020-01-01T00:00:00 # The Minimum Valid Time for this Mission (in ISO8601 format)
    max_valid_time: now # The Maximum Valid Time for this Mission (in ISO8601 format, or "now" for the current time)
    valid_data_levels: # List of Valid Data Levels for this Mission
      - l0
      - ql
      - l1
      - l2
      - l3
    instruments: # List of Instruments for this Mission
      - name: MyInstrument1 # The name of the instrument
        instrument_package: demo_myinst1 # The name of the instrument package (if different from the {mission}_{instrument} name)
        shortname: myinst # The short name of the instrument
        fullname: My Instrument Full Name # The full name of the instrument (to be used in plots, metadata, etc)
        targetname: MYINST # The target name of the instrument (to be used in file paths, etc)
      - name: MyInstrument2
        shortname: myinst2
        fullname: My Instrument 2 Full Name
        targetname: MYINST2
        extra_inst_names: # contains extra names that can be used to refer to this instrument (e.g. in file paths, etc)
          - myinst2_altname1
          - myinst2_altname2
        file_rules: # A list of rules for how to interpret files for this instrument. 
          # Raw Data (specific to .dat files)
          - levels: [raw] # The data levels that this rule applies to (can be a list of levels)
            extension: .dat # The file extension that this rule applies to
            time_format: "%y%m%d%H%M%S" # The time format to use for parsing the time from the file name (using strptime format codes)
          # Processed Files (Science / Standard Format Files)
          - levels: [l0, l1, ql, l2, l3] # The data levels that this rule applies to (can be a list of levels)
            extension: .fits # The file extension that this rule applies to
            time_format: "%Y%m%dT%H%M%S" # The time format to use for parsing the time from the file name (using strptime format codes)

  hermes:
    file_extension: cdf
    min_valid_time: 2020-01-01T00:00:00
    max_valid_time: now
    valid_data_levels:
      - l0
      - l1
      - ql
      - l2
      - l3
    instruments:
      - name: eea
        shortname: eea
        fullname: Electron Electrostatic Analyzer
        targetname: EEA
      - name: nemisis
        shortname: nem
        fullname: Noise Eliminating Magnetometer Instrument in a Small Integrated System
        targetname: NEM
      - name: merit
        shortname: mrt
        fullname: Miniaturized Electron pRoton Telescope
        targetname: MERIT
      - name: spani
        shortname: spn
        fullname: Solar Probe Analyzer for Ions
        targetname: SPANI
  
  padre:
    file_extension: fits
    min_valid_time: 2020-01-01T00:00:00
    max_valid_time: now
    valid_data_levels:
      - raw
      - l0
      - l1
      - ql
      - l2
      - l3
    instruments:
      - name: meddea
        shortname: meddea
        fullname: MeDDEA
        targetname: MEDDEA
        extra_inst_names:
          - mda
          - mdu
        file_rules:
          # Raw Data (specific to .dat files)
          - levels: [raw]
            extension: .dat
            time_format: "%y%m%d%H%M%S"
          # Processed Files (Science / Standard Format Files)
          - levels: [l0, l1, ql, l2, l3]
            extension: .fits
            time_format: "%Y%m%dT%H%M%S"
      - name: sharp
        shortname: sharp
        fullname:  Solar HARd X-ray Polarimeter
        targetname: SHARP
        extra_inst_names:
          - sp
          - shp
      - name: craft
        shortname: craft
        fullname: PADRE Spacecraft
        targetname: get
        extra_inst_names:
          - get
        file_rules:
          # Raw Data (specific to .csv files)
          - levels: [raw]
            extension: .csv
            time_format: "unix_ms"


  swxsoc_pipeline: # A special "mission" for SWxSOC affiliated instruments. 
  # This is a collection of multuple instruments and missions grouped together for convenience.
    file_extension: cdf
    valid_data_levels:
      - raw
      - l1c
      - l2
      - l3
    instruments:
      - name: reach
        shortname: reach
        fullname: REACH
        targetname: REACH
        instrument_package: swxsoc_reach
        file_rules:
          # SWxSOC Input Data (CSV & JSON Files) is already processed to a Level 1C
          - levels: [raw]
            extension: .csv
            time_format: "%Y%m%dT%H%M%S"
          - levels: [raw]
            extension: .json
            time_format: "%Y%m%dT%H%M%S"

  impax: # The Imaging Microburst Precipitation with Atmospheric X-ray emissions (IMPAX) CubeSat
    file_extension: cdf
    min_valid_time: 2026-01-01T00:00:00
    max_valid_time: now
    valid_data_levels:
      - raw
      - l0
      - l1
      - ql
      - l2
      - l3
    instruments:
      - name: iaxis
        shortname: iaxis
        fullname: IAXIS
        targetname: IAXIS
        instrument_package: impax_iaxis
      - name: ifire
        shortname: ifire
        fullname: IFIRE
        targetname: IFIRE
        instrument_package: impax_ifire
      - name: craft
        shortname: craft
        fullname: IMPAX Spacecraft
        targetname: IMPAX
        instrument_package: impax_craft

logger:
  # Threshold for the logging messages. Logging messages that are less severe
  # than this level will be ignored. The levels are 'DEBUG', 'INFO', 'WARNING', 'ERROR'
  log_level: INFO

  # Whether to use color for the level names
  use_color: true

  # Whether to log warnings.warn calls
  log_warnings: true

  # Whether to log exceptions before raising them
  log_exceptions: true

  # Whether to always log messages to a log file
  log_to_file: true

  # The file to log messages to
  log_file_path: swxsoc.log

  # Threshold for logging messages to log_file_path
  log_file_level: INFO

  # Format for log file entries
  log_file_format: "%(asctime)s, %(origin)s.%(funcName)s():%(lineno)d, %(levelname)s, %(message)s"