oai.content_dm.csu_sac validation results: [base contentdm is_shown_by logic & type vocab]

[christinklez]

Fixes requested

base contentdm mapper issue

• is_shown_by: For collection #26723 only. There is a URL transformation that needs to happen at the base contentdm mapper level. Need to fish <dc:identifier> and transform that URL; see the legacy base contentdm mapper: https://github.com/calisphere-legacy-harvester/dpla-ingestion/blob/cfe3dcb06008c0c6cb9d8207fe28bfaa1a855e4f/lib/mappers/contentdm_oai_dc_mapper.py#L31
  For example:
  Fish for: <dc:identifier>http://csus.contentdm.oclc.org/cdm/ref/collection/teaware/id/230</dc:identifier>
  & transform to: http://csus.contentdm.oclc.org/utils/getthumbnail/collection/teaware/id/230

type vocab additions

• type: type is not mapping, due to the wide range of type vocabulary terms provided. See notes in the comments, below. Type vocab additions requested in Address type enrichments (upper/lowercase normalization; dictionary) #668.

---

Resolved issues

• fetched, mapped, and actual (CSU DAMS records) do not line up. The Solr and Rikolti counts do not line up. To make things stranger, neither of these match up with the expected number of records in the CSU DAMS. See logs: https://7a8067cb-3b99-477e-a883-7e311175a9b4.c3.us-west-2.airflow.amazonaws.com/dags/validate_by_mapper_type/grid?dag_run_id=manual__2023-11-20T21%3A47%3A47%2B00%3A00&task_id=map_endpoint_task&tab=logs -- this is resolved via the fix to address empty mapped metadata pages.

• language: this is a question #11068 Rikolti is mapping language as "['jpn, fre, eng']" -- vernacular has dc:languagejpn, fre, eng</dc:language> (instead of three distinct entries) -- okay to accept? but there may be a related enrichment issue with languages. -- this is a question we should ask the contributor.
  

[christinklez]

Collection registry #: 11068

CSU SAC validation reports are missing types; it seems it is because there is a wide array of terms they are using for "type," and we do not have everything included in our type vocab set. There are 8 pages (0-8) of vernacular data; I went through pages 0-5; then skimmed 6-7. I think there are a handful we should consider including, but I think we need to really be okay with losing some of these. OR, we can opt to use the Registry "fill" option with "physical object", which it looks like many of these are.

Here are the ones I think we should consider adding:
<dc:type>Photographic print.</dc:type> --> image
<dc:type>Object sculpture</dc:type> --> physical object
<dc:type>Religious objects</dc:type> --> physical object
<dc:type>Ceremonial objects</dc:type> --> physical object
<dc:type>Postcards</dc:type> --> image
<dc:type>Paintings</dc:type> --> image
<dc:type>Animal paintings</dc:type> --> image
<dc:type>Poems</dc:type> --> text
<dc:type>Letters (signs)</dc:type> --> text

Here's my compiled running list of <dc:type> entries -- the bulk of which seem to be physical objects.

page 4:

<identifier>oai:csus.contentdm.oclc.org:jaac/897</identifier><dc:type>Game pieces</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/899</identifier><dc:type>Watercolors</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/918</identifier><dc:type>Lapel pins</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/919</identifier><dc:type>Brooches</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/920</identifier><dc:type>Shell (animal material)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/922</identifier><dc:type>Stuffed toys</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/923</identifier><dc:type>Object sculpture</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/924</identifier><dc:type>Necklaces</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/925</identifier><dc:type>Arrowheads</dc:type>

page 5:

<identifier>oai:csus.contentdm.oclc.org:jaac/1018</identifier><dc:type>Bowls (vessels)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1019</identifier><dc:type>Ladles</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1020</identifier><dc:type>Turners (culinary utensils)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1021</identifier><dc:type>Cooking baskets</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1022</identifier><dc:type>Mortars (grinding tools)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1023</identifier><dc:type>Pestles</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1024</identifier><dc:type>Inkstones</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1025</identifier><dc:type>Handscrolls</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1026</identifier><dc:type>Chopsticks</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1027</identifier><dc:type>Artists&#x27; brushes</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1028</identifier><dc:type>Charms</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1029</identifier><dc:type>Wallets</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1031</identifier><dc:type>Dolls</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1032</identifier><dc:type>Abici (calculators)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1033</identifier><dc:type>Carvings</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1036</identifier><dc:type>Ashtrays</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1037</identifier><dc:type>Wall hangings</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1039</identifier><dc:type>Vases</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1040</identifier><dc:type>Boxes (containers)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1041</identifier><dc:type>Postcards</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1042</identifier><dc:type>Paintings</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1045</identifier><dc:type>Animal paintings</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1049</identifier><dc:type>Poems</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1052</identifier><dc:type>Letters (signs)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1056</identifier><dc:type>Earrings</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1064</identifier><dc:type>Medals</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1065</identifier><dc:type>Boxes (containers)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1069</identifier><dc:type>Ornaments</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1071</identifier><dc:type>Buttons (information artifacts)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1074</identifier><dc:type>Toys (recreational artifacts)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1078</identifier><dc:type>Containers</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1087</identifier><dc:type>Decorations (honor)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1093</identifier><dc:type>Permits</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1094</identifier><dc:type>Legal notices</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1099</identifier><dc:type>Pedestals</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1104</identifier><dc:type>Tins</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1105</identifier><dc:type>Eggbeaters</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1108</identifier><dc:type>Plates (dishes)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1110</identifier><dc:type>Traditional medicine</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1112</identifier><dc:type>Religious objects</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1122</identifier><dc:type>Talismans; Dolls</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1133</identifier><dc:type>Balls (object genres)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1142</identifier><dc:type>Figurines</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1147</identifier><dc:type>Prayer books</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1148</identifier><dc:type>Bookstands</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1152</identifier><dc:type>Trophies (objects)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1153</identifier><dc:type>Holders</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1154</identifier><dc:type>Pens</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1156</identifier><dc:type>Pendants (jewelry)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1157</identifier><dc:type>Pins (jewelry)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1163</identifier><dc:type>Mailboxes</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1167</identifier><dc:type>Paperweights</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1173</identifier><dc:type>Insignias</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1174</identifier><dc:type>Patches (costume accessories)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1181</identifier><dc:type>Buttons (fasteners)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1183</identifier><dc:type>Billhooks</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1184</identifier><dc:type>Steamers (cookware)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1201</identifier><dc:type>Character dolls</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1202</identifier><dc:type>Caskets (personal gear)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1214</identifier><dc:type>Object sculpture</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1225</identifier><dc:type>Baskets</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1226</identifier><dc:type>Knives (culinary tools)</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1228</identifier><dc:type>Ashtrays</dc:type>
<identifier>oai:csus.contentdm.oclc.org:jaac/1231</identifier><dc:type>Fossils</dc:type>

page 6: (there are a ton more on this page)

page 7: (and there are more on this page)
<dc:type>Ceremonial objects</dc:type>

[barbarahui]

Fixes for is_shown_by logic should trickle down from the base OAI mapper. See #678

[christinklez]

Revisit type issues as part of #1145. Closing this as completed by #678