Sources available in LAT data server, there counterpart 4FGL name is missing or not working in Fermipy.

[nyctophile-aayu]

I am reporting an issue for a few AGN sources whose 4FGL source names are neither accepted in the Fermipy script nor available in the Fermi or Simbad catalog.

I have tried running the source PMN J0948+0022, and it goes through when the config file is read but doesn't run when its 4FGL name is provided while optimizing the source or later. I have checked the source name and its RA and DEC in the Fermi-LAT data server and Simbad catalog, and they match.

I have tried running the Fermipy script for a few other sources, but they don't go through the config file reading step. Also, their 4FGL name is missing even after their data is available in the Fermi-LAT server. These sources are:

1633+382,
0208-512,
NGC 5283,
Messier 81,
Messier 88,
Messier 106,
NGC 4725,
NGC 3147,
NGC 6814,
NGC 424,
NGC 3783,
NGC 4151,
4C +01.30,
NGC 326,
3C 123,
3C 6.1,
3C 109,
and the list goes on.

Not only these, but there are more sources whose 4FGL name is missing, and their lightcurve is also not updated on the Fermi webpage.

Could you please look into this and help me find a way to resolve this issue and where to find their corresponding 4FGL name?

I am sharing the relevant config files and fermipy logs.

bug.zip

[nyctophile-aayu]

Dear Team,

Please reply to the issue posted. It has been a couple of weeks since I posted the query. It is important to my work and the sources I need to analyze.

-Ayush

[zhipzhang]

Hi, I also found this problem. You can solve it by passing the name of xml files in catalogs. Usage:
model:
galdiff: '$FERMI_DIR/refdata/fermi/galdiffuse/gll_iem_v07.fits'
isodiff: '$FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R3_SOURCE_V3_v1.txt'
catalogs: 'gll_psc_v32.xml'

[nyctophile-aayu]

Hi @zhipzhang Can you comment more in detail? How can I pass the name and in which XML file ? (There are more than 1 XML files generate for a source.)

[MiltosMichailidis]

Hi,
Thank you for reaching out.

To ensure a comprehensive understanding of your issue, allow me to elucidate the following points:
The LAT data server accepts objects even in the absence of a corresponding 4FGL source (i.e., the object is undetected in Fermi data). Upon examination of a few objects in your list, I have seen that there is no corresponding 4FGL source. Consequently, if you wish to conduct such an analysis, it is necessary to incorporate a new source into your model (under any designation you prefer) and proceed with the analysis. This source can be added either in your configuration file or subsequently in your script. To incorporate such a source directly from the configuration file, rather than passing it as a target, it is necessary to define "sources" in the model section and provide the corresponding parameters. For further details, please refer to: https://fermipy.readthedocs.io/en/latest/config.html

Regarding the catalogs option in the model section of your configuration file:
This option defines a list of catalogs that will be merged to form a master analysis catalog from which sources will be drawn. Valid entries in this list can be FITS files or XML model files. Such files can be obtained from the Fermi webpages, specifically at: https://fermi.gsfc.nasa.gov/ssc/data/access/lat/14yr_catalog/
However, ['4FGL-DR3'] should work.

Have you identified any sources that are known GeV gamma-ray emitters but are not included in 4FGL?

[nyctophile-aayu]

Hi @MiltosMichailidis! I have tried the way you suggested but it is still generating the error at ROI fit. I have tried changing the name of the target as the 4FGL name too, but it also, fails at the same step showing error:

fit1 = gta.fit()

2024-12-17 15:48:49 INFO    GTAnalysis.fit(): Starting fit.
2024-12-17 15:48:51 WARNING GTAnalysis.profile_norm(): Source model for source 4FGL J1230.8+1223 has no counts in ROI
2024-12-17 15:48:51 WARNING GTAnalysis.profile_norm(): Upper limit not found.  Refitting normalization.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[51], line 1
----> 1 fit1 = gta.fit()

File ~/miniforge3/envs/fermipy/lib/python3.9/site-packages/fermipy/gtanalysis.py:3074, in GTAnalysis.fit(self, update, **kwargs)
   3072     if len(freePars) == 0:
   3073         continue
-> 3074     self.update_source(name, reoptimize=config['reoptimize'])
   3076 # Update roi model counts
   3077 self._update_roi()

File ~/miniforge3/envs/fermipy/lib/python3.9/site-packages/fermipy/gtanalysis.py:4106, in GTAnalysis.update_source(self, name, paramsonly, reoptimize, **kwargs)
   4103 npts = self.config['gtlike']['llscan_npts']
   4104 optimizer = kwargs.get('optimizer', self.config['optimizer'])
-> 4106 sd = self.get_src_model(name, paramsonly, reoptimize, npts,
   4107                         optimizer=optimizer)
   4108 src = self.roi.get_source_by_name(name)
   4109 src.update_data(sd)

File ~/miniforge3/envs/fermipy/lib/python3.9/site-packages/fermipy/gtanalysis.py:4264, in GTAnalysis.get_src_model(self, name, paramsonly, reoptimize, npts, **kwargs)
   4261     pass
   4262 # self.logger.error('Failed to update source parameters.',
   4263 #  exc_info=True)
-> 4264 lnlp = self.profile_norm(name, savestate=True,
   4265                          reoptimize=reoptimize, npts=npts,
   4266                          optimizer=optimizer)
   4268 src_dict['loglike_scan'] = lnlp['loglike']
   4269 src_dict['dloglike_scan'] = lnlp['dloglike']

File ~/miniforge3/envs/fermipy/lib/python3.9/site-packages/fermipy/gtanalysis.py:2472, in GTAnalysis.profile_norm(self, name, logemin, logemax, reoptimize, xvals, npts, fix_shape, savestate, **kwargs)
   2469         else:
   2470             xvals = np.linspace(0, lims['ul'], npts)
-> 2472 o = self.profile(name, parName,
   2473                  reoptimize=reoptimize, xvals=xvals,
   2474                  savestate=savestate, **kwargs)
   2476 if savestate:
   2477     saved_state.restore()

File ~/miniforge3/envs/fermipy/lib/python3.9/site-packages/fermipy/gtanalysis.py:2684, in GTAnalysis.profile(self, name, parName, logemin, logemax, reoptimize, xvals, npts, savestate, **kwargs)
   2681         xvals = val * 10 ** xvals
   2683 if np.isnan(xvals).any():
-> 2684     raise RuntimeError(
   2685         "Parameter scan points for %s::%s include infinite value." % (name, parName))
   2687 # Update parameter bounds to encompass scan range
   2688 try:

RuntimeError: Parameter scan points for 4FGL J1230.8+1223::norm include infinite value.

My updated config file for this code was:

data:
  evfile : events.txt
  scfile : SC00.fits

binning:
  roiwidth   : 20.0
  binsz      : 0.1
  binsperdec : 8

selection :
  emin : 100
  emax : 300000
  zmax    : 90
  evclass : 128
  evtype  : 3
  tmin    : 239557417
  tmax    : 733622405
  filter  : 'DATA_QUAL>0 && LAT_CONFIG==1'
  #target : 'Messier 87'

gtlike:
  edisp : True
  edisp_bins: -1
  irfs : 'P8R3_SOURCE_V3'
  edisp_disable: ['isodiff']

model:
  src_roiwidth : 20.0
  galdiff  : '$FERMI_DIFFUSE_DIR/gll_iem_v07.fits'
  isodiff  : 'iso_P8R3_SOURCE_V3_v1.txt'
  catalogs : ['4FGL-DR3']
  sources :
    - { 'name' : '4FGL J1230.8+1223', 'ra' : 187.706, 'dec' : 12.3911, 'SpectrumType' : LogParabola }

components:
  - { selection : { evtype : 4  } } # PSF0
  - { selection : { evtype : 8  } } # PSF1
  - { selection : { evtype : 16 } } # PSF2
  - { selection : { evtype : 32 } } # PSF3

fileio:
   outdir : data

Please suggest what else can I try?

[MiltosMichailidis]

Hi,
Before you do anything else, let's clear up the confusion here. The "sources" option in the model section of your configuration file should only be used to add additional point-source or extended components beyond those already defined in the master catalog. M87 is a gamma-ray source detected by Fermi-LAT, and the corresponding 4FGL source is already included in the master file you're using for your analysis. There's no need to try adding a source that already exists.

If you add a source at the same location as an existing 4FGL source, you won't get any meaningful results. To analyze M87, simply include it in the "target" option of the selection section. The setup, optimization, and fitting should then run smoothly, and you can continue to refer to the target by its name (no need to switch to the 4FGL name) when computing maps or any other products.

If the fit task continues to fail, there may be an issue, such as too many free parameters or some other problem. This depends on how you set up your analysis script. But the key here is to avoid redundantly adding sources that are already accounted for in the master catalog.

The error you're seeing now is due to the incorrect analysis approach you're using. Please try again following these guidelines, and let me know how it goes.