-
Notifications
You must be signed in to change notification settings - Fork 1
slac-epics/bsaDriver
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
#============================================================== # # Name: bsaDriver # # Rem: # EPICS driver module # Superconducting Timing System # # Auth: 17-Mar-2017, Kukhee Kim (KHKIM) # Rev: 27-Sep-2022, Dawood Alnajjar (DNAJJAR) # Rev: 22-Mar-2023, Marcio Donadio (MARCIO) #============================================================== This is an epics driver module to provide BSA, BSSS, BSAS, and BLD interface for superconducting timing High Performance System (HPS) which is based on ATCA platform. We are calling BSA, BSSS, BSAS, and BLD as "acquisition services" in this document. Important definition: - "Signal" is used in this document as a piece of data calculated by the system and exported to the acquisition services. Example: X, Y, and TMIT are BPM signals. Originally the word "channel" was used with this meaning but recently this was changed. The software was all built using the "channel" word to define this concept, so you will see IOC shell commands, reports, and database record names using "channel" instead of "signal". The change in software names will require more time to be implemented. This module provides: - Epics IOC shell commands to configure/setup all the acquisition services. - Epics database templates for: - BSA and BSSS PVs. - Diagnostic PVs for all services. - Control PVs for BSSS, BSAS, and BLD. - pvxs PVs for BSAS. ******************************************************************************* *** 1. Building the IOC application with the module *** ******************************************************************************* 1.1 - Packages for configure/CONFIG_SITE: . cpsw/framework R4.4.1 . yaml-cpp-0.5.3_boost-1.64.0 . boost 1.64.0 . timing/bsa R2.4.0 configure/CONFIG_SITE should have the following*: CPSW_FRAMEWORK_PACKAGE_NAME=cpsw/framework CPSW_FRAMEWORK_VERSION=R4.4.1 CPSW_FRAMEWORK_TOP=$(PACKAGE_SITE_TOP)/$(CPSW_FRAMEWORK_PACKAGE_NAME)/$(CPSW_FRAMEWORK_VERSION) CPSW_FRAMEWORK_LIB = $(CPSW_FRAMEWORK_TOP)/$(PKG_ARCH)/lib CPSW_FRAMEWORK_INCLUDE = $(CPSW_FRAMEWORK_TOP)/$(PKG_ARCH)/include YAML_NAME=yaml-cpp YAML_VERSION=yaml-cpp-0.5.3_boost-1.64.0 YAML_TOP=$(PACKAGE_SITE_TOP)/$(YAML_PACKAGE_NAME)/$(YAML_VERSION) YAML_LIB= $(YAML_TOP)/$(PKG_ARCH)/lib YAML_INCLUDE=$(YAML_TOP)/$(PKG_ARCH)/include BOOST_PACKAGE_NAME=boost BOOST_VERSION=1.64.0 BOOST_TOP=$(PACKAGE_SITE_TOP)/$(BOOST_PACKAGE_NAME)/$(BOOST_VERSION) BOOST_LIB = $(BOOST_TOP)/$(PKG_ARCH)/lib BOOST_INCLUDE = $(BOOST_TOP)/$(PKG_ARCH)/include BSA_PACKAGE_NAME=timing/bsa BSA_VERSION=R2.4.0 BSA_TOP=$(PACKAGE_SITE_TOP)/$(BSA_PACKAGE_NAME)/$(BSA_VERSION) BSA_LIB=$(BSA_TOP)/$(PKG_ARCH)/lib BSA_INCLUDE=$(BSA_TOP)/$(PKG_ARCH)/include * We are assuming that files configure/CONFIG_SITE.Common.<architecture> defining $(PKG_ARCH) already exist as this is standard for all IOC applications using LinuxRT. 1.2 - EPICS modules for configure/RELEASE . yamlLoader/R2.2.0 . asyn/R4.39-1.0.1 . pvxs/R0.3.1-0.0.1 . bsaDriver/R3.1.0 ASYN_MODULE_VERSION=R4.39-1.0.1 YAMLLOADER_MODULE_VERSION=R2.2.0 BSADRIVER_MODULE_VERSION=R3.1.0 PVXS_MODULE_VERSION=R0.3.1-0.0.1 ASYN=$(EPICS_MODULES)/asyn/$(ASYN_MODULE_VERSION) YAMLLOADER=$(EPICS_MODULES)/yamlLoader/$(YAMLLOADER_MODULE_VERSION) PVXS=$(EPICS_MODULES)/pvxs/$(PVXS_MODULE_VERSION) BSADRIVER=$(EPICS_MODULES)/bsaDriver/$(BSADRIVER_MODULE_VERSION 1.3 - src/Makefile: # ===================================================== # Path to "NON EPICS" External PACKAGES: USER INCLUDES # ===================================================== USR_INCLUDES += -I$(BOOST_INCLUDE) USR_INCLUDES += -I$(CPSW_FRAMEWORK_INCLUDE) USR_INCLUDES += -I$(YAML_INCLUDE) USR_INCLUDES += -I$(BSA_INCLUDE) # ===================================================== #====================================================== # PATH TO "NON EPICS" EXTERNAL PACKAGES: USER LIBRARIES # ====================================================== boost_system_DIR = $(BOOST_LIB) cpsw_DIR = $(CPSW_FRAMEWORK_LIB) yaml-cpp_DIR = $(YAML_LIB) bsa_DIR = $(BSA_LIB) # ====================================================== # ====================================================== # LINK "NON EPICS" EXTERNAL PACKAGE LIBRARIES STATICALLY # ======================================================= USR_LIBS_Linux += boost_system USR_LIBS_Linux += cpsw USR_LIBS_Linux += yaml-cpp USR_LIBS_Linux += bsa # ====================================================== # RHEL6 is too old and its gcc is not compatible with pvxs, which is needed by # BSAS. So we need this to build only for linuxRT. ifeq ($(T_A),linuxRT-x86_64) PROD_IOC = <appliccation> endif <application>_DBD += asyn.dbd <applicaiton>_DBD += yamlLoader.dbd <application>_DBD += bsaAsynDriver.dbd <application>_LIBS += cpsw <application>_LIBS += bsa <applicaiton>_LIBS += asyn <application>_LIBS += yamlLoader <application>_LIBS += bsaDriver # EPICS 7 PVXS support # Order of the two libraries is important: <application>_LIBS += pvxsIoc <application>_LIBS += pvxs <application>_DBD += pvxsIoc.dbd 1.4 - Db/Makefile. Add the files only for the services that you need to use. DB_INSTALLS += $(BSADRIVER)/db/bsa.db DB_INSTALLS += $(BSADRIVER)/db/bsss.db DB_INSTALLS += $(BSADRIVER)/db/bsssCtrl.db DB_INSTALLS += $(BSADRIVER)/db/bsas.db DB_INSTALLS += $(BSADRIVER)/db/bsasCtrl.db DB_INSTALLS += $(BSADRIVER)/db/bld.db DB_INSTALLS += $(BSADRIVER)/db/bldCtrl.db - bsa.db ; BSA database template for each BSA signal. - bsss.db ; BSSS database template for each BSSS signal. - bsas.db ; BSAS database template for enabling/disabling and defining severity filters per signal. - bld.db ; BLD database template for enabling/disabling and defining severity filters per signal. - bsssCtrl.db ; BSSS controls database template - bsasCtrl.db ; BSAS controls database template - bldCtrl.db ; BLD controls database template ******************************************************************************* *** 2. Basic commands required for st.cmd *** ******************************************************************************* 2.1 - yamlLoader Driver: The yamlLoader driver needs to load up the hierarchy file before BSA initialization. Example: # cpswLoadYamlFile(<top level yaml path>, "NetIODev, "", <carrier board IP) cpswLoadYamlFile("000TopLevel.yaml", "NetIODev", "", "10.0.3.105") Please, check up more details in the README file of yamlLoader driver. 2.2 - Load YAML configuration file to disable all acquisition services. A file template with the YAML contents is available at README.disable_bld_bsss_bsas. The line below needs to be added right after cpswLoadYamlFile. Please, adjust the path for disable_bld_bsss_bsas.yaml accordingly. The second argument, which is the yaml path may be different from application to application. The one below should work on most cases. If it doesn't work for you, you need to contact the firmware engineer to get the correct one. # Load configuration to disable BSA/BLD/BSSS/BSAS on boot cpswLoadConfigFile("${YAML_DIR}/config/disable_bld_bsss_bsas.yaml", "mmio/AmcCarrierCore/AmcCarrierBsa") 2.3 Adding BSA names for the driver: Before starting this configuration, you need to receive from the firmware engineer the list of BSA signals available, with its data type. Make sure to have this list in the correct order. When asking the firmware engineer, use the following jargon: "I need the ordered list of data that goes into the diagnostic bus, with the data type". bsaDriver provides the ioc shell command bsaAdd(<BSA key>, <signal data type>) You need to use bsaAdd for each of the BSA signals from the list provided by the firmware engineer, in the same order as the list. <BSA key> can be any string you want to describe the signal. <signal data type> can be: "int32", "uint32", "float32". Details on data type restrictions are available later in a specific section of the README. Optionally, should the signal stay untouched (not scaled using slope/offset or casted to another type), a third parameter is passed to the bsaAdd function. This parameter is only used by BLD, not BSA/BSAS/BSSS. More about slope and offset will be explained later. - "0" (or empty) -> Allowed to apply slope/offset - "1" -> Do not touch *Important note!* Although the command is called bsaAdd, the configuration you make by using it is going to be used by all acquisition services. The driver also provides the IOC shell command "bsaList()" to print out the BSA name list, so the IOC engineer can verify the BSA names loaded. Example of use of the bsaDriver functions: # ==================================== # Setup BSA Driver # ==================================== # We need to add the status signals because they are the first ones, even if # they are not used by the acquisition services. bsaAdd("STATUSAMC0", "uint32", 1) bsaAdd("STATUSAMC1", "uint32", 1) # Add signals for BSA, BSSS, BSAS bsaAdd("TMITAMC0", "int32") bsaAdd("TMITAMC1", "int32") bsaAdd("XFIXEDPAMC0", "int32") bsaAdd("XFIXEDPAMC1", "int32") bsaAdd("YFIXEDPAMC0", "int32") bsaAdd("YFIXEDPAMC1", "int32") # Add signals for BLD bsaAdd("TMITFLOATAMC0", "float32", 1) bsaAdd("TMITFLOATAMC1", "float32", 1) bsaAdd("XFLOATPAMC0", "float32", 1) bsaAdd("XFLOATPAMC1", "float32", 1) bsaAdd("YFLOATPAMC0", "float32", 1) bsaAdd("YFLOATPAMC1", "float32", 1) sioc-b84-ts01> bsaList() Total 14 BSA(s) has(have) been registered 0 STATUSAMC0, uint32 1 STATUSAMC1, uint32 2 TMITAMC0, int32 3 TMITAMC1, int32 4 XFIXEDPAMC0, int32 5 XFIXEDPAMC1, int32 6 YFIXEDPAMC0, int32 7 YFIXEDPAMC1, int32 8 TMITFLOATAMC0, float32 9 TMITFLOATAMC1, float32 10 XFLOATPAMC0, float32 11 XFLOATPAMC1, float32 12 YFLOATPAMC0, float32 13 YFLOATPAMC1, float32 2.3.1 Adding dependent secondary BSA names for the driver The secondary BSA node shares BSA data source with the primary. The secondaries can have different data conversion or separate slope and offset from the primary BSA. The idea is that you can provide different conversions for each signal. Example: the signal related to the charge in the BCM could be presented as coulombs, nano coulombs, or number of electrons. After creating the primary BSA node using bsaAdd, create the secondary BSA node as the following: bsaAddSecondary(<BSA key for primary>, <BSA key for secondary>, <data type>) - The "BSA key for primary" should exist before adding the secondary - The "BSA key for secondart" will be used as the BSAKEY database macro for dbLoadRecords(). More about this will be explained in other README specific files. Example: bsaAddSecondary("TMITAMC0", "TMIT0C0", "int32") bsaAddSecondary("TMITAMC0", "TMIT0C1", "int32") bsaAddSecondary("TMITAMC0", "TMIT0C2", "int32") bsaAddSecondary("TMITAMC0", "TMIT0C3", "int32") bsaAddSecondary("TMITAMC0", "TMIT0C4", "int32") bsaAddSecondary("TMITAMC0", "TMIT0C5", "int32") In this example, we added 6 secondary BSAs for the TMITAMC0 BSA. bsaList() Total 14 BSA(s) has(have) been registered 0 STATUSAMC0, uint32 1 STATUSAMC1, uint32 2 TMITAMC0, int32 +----- secondary node 0 TMIT0C0, int32 +----- secondary node 1 TMIT0C1, int32 +----- secondary node 2 TMIT0C2, int32 +----- secondary node 3 TMIT0C3, int32 +----- secondary node 4 TMIT0C4, int32 +----- secondary node 5 TMIT0C5, int32 3 TMITAMC1, int32 4 XFIXEDPAMC0, int32 5 XFIXEDPAMC1, int32 etc. bsaList() command shows the primary node and its secondary nodes. 2.3.2 More about signal Data Types bsaDriver and firmware allow {"int32", "uint32", "float32"} type of data. Due to limited "space" in the FPGA chip, the firmware will only calculate statistics (mean and rms) for a BSA signal if its type is int32. This programming is in the firmware and can't be configured by software. So, if the firmware engineer sent you the list of BSA signals and a few of them are float32 you already know that this signal will not have mean and rms calculated for BSA. Sometimes the signal will represent a status with each of the bits from a uint32 representing a condition. Of course a status can't be part of statistic calculation and this was also already programmed in the firmware. 2.4 Driver initialization For the driver initialization, bsaDriver provides the IOC shell command bsaAsynDriverConfigure(<asyn port name for BSA>, <path for register space>, <path for dram space>) The driver initialization command must be called after completing the yaml loading by yamlLoader driver (item 2.2) and after the BSA name list is built (item 2.3). <asyn port name for BSA> can be any string. This string will be used later to configure the other acquisition services and also for the BSA EPICS database template. The register path and dram path should be consistent with the yaml files. In the example below, ROOT path of hierarchy is NetIODev. The register space is located at NetIODev/mmio/AmcCarrierTimingGenerator/AmcCarrierCore/AmcCarrierBsa, and the stream dram is located at NetIODev/strm/AmcCarrierDRAM/dram. Example: bsaAsynDriverConfigure("bsaPort", "mmio/AmcCarrierCore/AmcCarrierBsa","BsaRam/dram") 2.4.1 Check 000TopLevel.yaml if bsaAsynDriverConfigure() fails to load bldstream needs to be defined in the top yaml file. Although it is called bldstream, it serves all acquisition services. # BLD Stream bldstream: &bldstream SRP: protocolVersion: SRP_UDP_NONE UDP: port: 8198 numRxThreads: *udpThreads threadPriority: *udpPri pollSecs: 1 # Inside NetIoDev session bldstream: class: Field at: <<: *bldstream 2.5 Enable/Disable BSA After the driver initialization, the BSA can be disabled and enabled anytime via the following commands: bsaAsynDriverEnable() bsaAsynDriverDisable() The bsaAsynDriverConfigure() command (item 2.4) enables BSA by default (backward compatible). The enable/disable was introduced for a work around solution for RSSI communication down issue. We found that RSSI communication was down when bsaDriver had a heavy communication load. The communication down caused exception on iocs. When the BSA driver is disabled, the firmware still works for BSA and keeps collecting data internally to the carrier board DRAM. On the other hand, the software does not check up new data and does not retrieve data from DRAM. Using bsaAsynDriverDisable is not harmful for the firmware operation. If the write pointer reach the boundary of the buffer, the pointer moves back to the start address of the buffer. 2.6 Linear Conversion Usually the signals arrive from the firmware as an integer number that needs to be converted to the correct engineering unit. The BSA database template provides slope and offset PVs for each BSA signal created by bsaAdd or bsaAddSecondary (bsakey). These PVs affect all acquisition services at the same time so the signal can be shown with the right unit everywhere. The linear conversion affects the following: bsa_data = slope * raw_bsa_data + offset bsa_rmsa_value = slope * raw_rms_value Slope and Offset PV names are: $(DEV):$(SECN)_SLOPE_BSA $(DEV):$(SECN)_OFFSET_BSA Macros: $(DEV) ; device PV name $(SECN) ; signal name as part of the PV name These are in the autosave list. The database template bsa.db has default value for SLOPE=1 and OFFSET=0. These will be overriden by autosave, once the user setting is saved. Example: if the signal name in the PV is TMIT (SECN), consider these default values for the PVs $(DEV):TMIT_SLOPE_BSA = 1 $(DEV):TMIT_OFFSET_BSA = 0 Suppose that the firmware collected the BSA buffer 27 with 3 values: $ caget $(DEV):TMITHST27 $(DEV):TMITHST27 3 1000 2000 3000 Now let's change slope and offset: $(DEV):TMIT_SLOPE_BSA = 0.1 $(DEV):TMIT_OFFSET_BSA = 32 The new result for the buffer will be: $ caget $(DEV):TMITHST27 $(DEV):TMITHST27 3 132 232 332 2.7 Callback Queue Size As bsaDriver running all services takes a lot of resources, we need to increase the size of the queue. Place this command anywhere before iocInit(): callbackSetQueueSize(12000) ******************************************************************************* *** 3. Diagnostic reports *** ******************************************************************************* To check the status of the acquisition services engine. Although the driver is called bldStreamDriver, this is the driver that deals with all 4 services: BSA, BSSS, BSAS, and BLD. Use this command on the IOC shell (if instead of 5 you use 10, some of the counters will be reset): epics> dbior bldStreamDriver 5 Driver: bldStreamDriver BLD Stream for root_0 read size : 48 read count : 44 bld count : 20 bsss count : 0 bsas count : 20 else count : 0 overrun : 0 bld callback : 0x741d40 bld_usr : 0x14909a0 bsss_callback: 0x741a60 bsss_usr : 0x104f6c0 bsas_callback: 0x747ce0 bsas_usr : 0x147dba0 free list : 0x1041ec0 bldQueue[0] name of thread: bldQ0_root_0 pending count : 0 water mark : 0 fail count : 0 overrun count : 0 bsssQueue[0] name of thread: bsssQ0_root_0 pending count : 0 water mark : 0 fail count : 0 overrun count : 0 bsasQueue[0] name of thread: bsasQ0_root_0 pending count : 0 water mark : 0 fail count : 0 overrun count : 0 -------------------------------- BSAS Packet: size(48) -------------------------------- timestamp (64bit) : 18d7b15752e3f pulse id (64bit) : 15ffdf0880 channel mask : 8 row number : 0 table_count : 6 edef_index : 0 byte pad : 0 service mask (0x80): 8 PL CH CNT EVL ESQ FIX VAL SUM SQUARE MIN MAX ------------------------------------------------------------------------- 0 3 1 N N N 38b 38b c8d79 38b 38b -------------------------------- BLD Packet: size(40) -------------------------------- timestamp : 18d7b15752e3f pulse ID : 15ffdf0880 channel mask : 88 service mask : 20000003 Chan. data found : 2 Severity mask : 3fffffff33333333 epics> dbior serviceAsynDriver 1 Driver: serviceAsynDriver named_root: root_0, port: bsssPort, driver instace: 0x104f6c0, number of channels: 14 named_root: root_0, port: bldPort, driver instace: 0x14909a0, number of channels: 14 ******************************************************************************* *** 4. Timing filters *** ******************************************************************************* In normal conducting timing, using the EVG (Event Generator), you would select the acquisition frequency by using EDEFs (Event Definitions). The correspondent of it for the superconducting timing TPG (Timing Pattern Generator) is called the timing filter. Timing filters are a way of configuring a frequency and applying a few filters to it, like the destination where the electron bunch is sent to. Timing filters are used not only by the acquisition services, but also by the configuration of triggers. So, explaining its configuration is out of scope of this README. There's a good introductory text on this Confluence page: https://confluence.slac.stanford.edu/x/4wVwFg. Check for the section related to timing filters. ******************************************************************************* *** 5. Database macro conventions for all acquisition services *** ******************************************************************************* In all available READMEs, when describing EPICS databases, we mention a DEVICE_PREFIX and a TPR_PREFIX. By DEVICE_PREFIX we mean the traditional combination of DEVICE:AREA:LOCA. Some systems may contain multiple DEVICE_PREFIX in the same IOC. Say the MPS system, for example. Below is a real list of DEVICE_PREFIX for one IOC in production: CBLM:UNDH:4575 CBLM:UNDH:4676 BEND:DMPH:395 PBLM:DMO0:846:AH PBLM:LTUS:846:AH PBLM:LTUH:846:AH bsaDriver provides a few EPICS database files which generate PVs that are unique for the entire IOC. In this case the question is inevitable? Which prefix should I use? CBLM:UNDH:4575? PBLM:DMO0:846:AH? You see that none of them makes sense. For these cases a new prefix is needed. One that is unique throughout the IOC. We are calling it TPR_PREFIX. MPS could use, for example, TPR:DMO0:MP01:1 ******************************************************************************* *** 6. Severity masks *** ******************************************************************************* Some of the databases provide a way of setting a severity mask. Signals coming from the firmware are classified according to a severity, similiar to alarm severities from EPICS. Each severity type has a weight: None < Minor < Major < Invalid For all services you can determine that data with a given severity is discarded. If the severity of the data is less than or equal to the chosen severity mask, the data is accepted by the acquisition service. Otherwise, it is thrown away. In summary, the rule is: Data accepted if: data severity in firmware <= severity mask Example 1: Data in firmware is classified as Major. Severity mask is None. Major > None ==> data rejected Example 2: Data in firmware is classified as Minor. Severity mask is Invalid. Minor <= Invalid ==> data accepted In other words, if Invalid is used for the severity mask, any data will be accepted. ******************************************************************************* *** 7. Configuration of RT priorities for SLAC protocols and network stack *** ******************************************************************************* The 000TopLevel.yaml file must appropriately set the real-time priorities for the SLAC protocols used to communicate with the ATCA board. If the configuration is not set correctly, the IOC may crash when accessing the acquisition services. Check if the yaml file contains this information: connectionParameters: udpPri: &udpPri 80 udpThreads: &udpThreads 4 rssi: &rssiParams retransmissionTimeoutUS: 100000 cumulativeAckTimeoutUS: 50000 threadPriority: 82 Along the entire yaml you should see these configurations for the threadPriority of UDP and RSSI: UDP: port: *udpPortNumber0 threadPriority: *udpPri RSSI: <<: *rssiParams threadPriority: 81 inpQueueDepth: 32 The cpu's st.cmd (previously known as "kernel-modules.cmd") must contain correct thread priorities and buffer size for the kernel network stack. This ensures that the network will not get stalled when the IOCs start to use a lot of CPU. Below is what is needed (in this example, eth5 is the interface connected to the ATCA crate). Each interface connected to the ATCA must have the priority thread set. # ================================================================== # escalating RT priority for network kernel threads (SCHED_FIFO, 84) # ================================================================= for PID in $(ps -elF | grep eth5 | grep -v grep | awk '{print $4}'); do chrt -f -p 84 $PID ; done # ====================================== # increasing receive buffer size to 10MB # ====================================== sysctl -w net.core.rmem_default=10485760 Also, for BLD to work properly, jumbo frames need to be configured for each of the participant network interfaces. This is also configured in the cpu's st.cmd: # Add jumbo frames to allow for proper BLD communication ifconfig eth5 mtu 9000 up ******************************************************************************* *** 8. Configuration of individual acquisition services *** ******************************************************************************* BSA, BSSS, BSAS, and BLD configuration are described in specific readme files: BSA - README.bsa_only BSSS - README.bsss_only BSAS - README.bsas_only BLD - README.bld_only ******************************************************************************* *** 9. Example st.cmd *** ******************************************************************************* An example st.cmd is available at README.example_st_cmd
About
EPICS driver module to provide BSA, BSSS, BSAS, and BLD interfaces for the superconducting timing High Performance System (HPS)
Resources
Stars
Watchers
Forks
Packages 0
No packages published