Handle custom fields in Citavi 5 XML.js #3183

zuphilip · 2023-11-09T18:32:58Z

In Citavi, it is possible to define and use custom fields (=Freitextfelder). These changes provide them as well in Zotero.

The import now saves the contents into separate notes prefixed with the label of custom fields. For example:

Custom fields (=Freitextfelder) in Citavi can be defined and used. This changes imports them as well in Zotero.

Also fixing potential problems handling keywords with other tags in general.

zuphilip · 2023-11-28T08:51:30Z

I have pushed now some further improvements to the Citavi translator. Most fixed issues are from the forum post https://forums.zotero.org/discussion/comment/449182/#Comment_449182 and the provided example. Additionally saving the BibTeX key was mentioned in another round and I picked this up to include it now in the official translator. It would be great to hear any feedback to this pull request.

zuphilip · 2024-01-18T08:15:40Z

Bump

It would be great if this could be discussed and merged. I know that there probably more and more people will need to migrate from Citavi as several contracts in Germany ended or will end soon. This PR improves several things which were reported from such people.

CC @adam3smith

adam3smith · 2024-01-21T18:41:25Z

This all looks good to me -- could we either get a test or at a minimum link to test data (ideally linked from a comment in the tranlsator)? That'd help hugely with maintenance.

AbeJellinek · 2024-01-22T22:24:31Z

Citavi 5 XML.js

+			var label = rememberCustomFields[field];
+			var customField = ZU.xpathText(references[i], './' + field);
+			if (customField) {
+				item.notes.push({ note: label + ": " + customField, tags: ["#CustomField"] });


We don't typically add automatic tags to notes AFAIK, especially with a hash prefix. Can we leave it out?

I believe we do for RIS and Endnote XML to allow quick filter and delete?

Similar tags in notes are used already in this Citavi translator itself: #Notes, #TableOfContents, #Evaluation, and #todo (from tasks imported as standalone notes). There are different fields in Citavi which are all mapped to notes in Zotero. With the tags one can still differentiate them and possibly also filter for them as @adam3smith suggested.

I think that this is especially useful for custom fields, which in some cases have a better field in Zotero to save the information. This now could then be realized by using a little script after the import translator here. For example I started with this adaptable proof-of-concept script for moving the #customFields notes to some other field within Zotero. Overall, one would then need to do the import first (which is the same for everyone), and possibly some individual script to move some notes further afterwards.

zuphilip · 2024-01-23T10:37:50Z

This all looks good to me -- could we either get a test or at a minimum link to test data (ideally linked from a comment in the translator)? That'd help hugely with maintenance.

The custom fields are already part of some of the saved test cases now. However, I can also check whether it is possible to provide another example covering the things here.

adam3smith · 2024-01-27T03:28:53Z

The custom fields are already part of some of the saved test cases now. However, I can also check whether it is possible to provide another example covering the things here.

But then they should up in changed tests, no? In that case, maybe just need to update the tests, or am I misunderstanding what you're saying?

zuphilip · 2024-01-29T08:07:58Z

But then they should up in changed tests, no? In that case, maybe just need to update the tests, or am I misunderstanding what you're saying?

No, the test cases are not part of the translator here. I think there was an parsing error when we first tried that, but I can't remember exactly because that was some years ago. Instead test data for Citavi translator is saved in some gists and then just linked here as comments in the translator:

translators/Citavi 5 XML.js

Lines 42 to 49 in d955310

    
           /* 
        
           TEST DATA can be found here: 
        
            - Single reference (162 KB) text: https://gist.github.com/zuphilip/02d6478ace4636e4e090e348443c551e 
        
            - Larger project (1221 KB): https://gist.github.com/zuphilip/76ce89ebbdac0386507b36cff3fd499a 
        
            - Other project (1,11 MB): https://gist.github.com/anonymous/10fc363b6d79dae897e296a4327aa707 
        
            - Citavi 6 project (935 KB): https://gist.github.com/zuphilip/00a4ec6df58ac24b68366e32531bae4b 
        
            - Nested categories: (34 KB): https://gist.github.com/tnajdek/b2375e52b48c7bf82f9f592b4f2122f5 
        
           */

JAC28

Proposal for further extension of the translator to take into account image notes.

JAC28 · 2024-10-11T10:04:19Z

Citavi 5 XML.js

 	importCategories({ categories, doc, progress });
 }

+function extractNote(doc, noteXML, rememberTags) {


One major drawback when import data from Citavi to Zotero is loosing all knowledge-items with images ("Bildzitat"). As far as I am aware, it is currently not possible to attach an image to a note, which results in many notes without much content after import. To simplify manual fixing afterward, I suggest tagging the note and include the filename into the note, as follows. The result could look like

JAC28 · 2024-10-11T10:05:40Z

Citavi 5 XML.js

+	noteObject.id = ZU.xpathText(noteXML, '@id');
+	var title = ZU.xpathText(noteXML, 'CoreStatement');
+	var text = ZU.xpathText(noteXML, 'Text');
+	var pages = extractPages(ZU.xpathText(noteXML, 'PageRange'));


Suggested change

var pages = extractPages(ZU.xpathText(noteXML, 'PageRange'));

var pages = extractPages(ZU.xpathText(noteXML, 'PageRange'));

var address = ZU.xpathText(citations[j], 'Address');

var hasMissingImage =false;

JAC28 · 2024-10-11T10:07:21Z

Citavi 5 XML.js

+	if (text) {
+		text = text.split(/\r?\n/).join("<br />");
+		noteObject.note += "<p>" + text + "</p>\n";
+	}


Add text to note if address is file name. Empty line before text to ease inserting of image later.

Suggested change

}

}

if (address) {

var addressMap = JSON.parse(address);

// Based on Demo project mapping is:

// 1->File

// 5->Link

if (addressMap.LinkedResourceType==1){

noteObject.note += "" + 'Missing Image with filename: ';

noteObject.note += "<pre><code>" + addressMap.UriString + "</code></pre>\n";

var hasMissingImage = true;

}

}

JAC28 · 2024-10-11T10:10:03Z

Citavi 5 XML.js

+		for (let j = 0; j < rememberTags[noteObject.id]; j++) {
+			noteObject.tags.append(rememberTags[noteObject.id][j]);
+		}
+	}


Finally, add a tag similar as #todo for easy filtering

Suggested change

}

}

if (hasMissingImage) {

noteObject.tags.push("#missingImage");

}

…ilenames

zuphilip added 5 commits November 9, 2023 19:23

Handle custom fields in Citavi 5 XML.js

3dd5f32

Custom fields (=Freitextfelder) in Citavi can be defined and used. This changes imports them as well in Zotero.

Citavi Import: Keep line breaks in knowledge items

fd344bc

Citavi Import: Add BibTeX key to extra field

1bf85f6

Citavi Import: Add keywords for knowledge elements

fded2ea

Also fixing potential problems handling keywords with other tags in general.

Citavi Import: Import standalone knowledge also

86db270

zuphilip force-pushed the citavi-custom-fields branch from 2cd7b19 to fc0978d Compare November 28, 2023 11:39

Citavi Import: Fix linting issues

06adfb9

zuphilip force-pushed the citavi-custom-fields branch from fc0978d to 06adfb9 Compare November 28, 2023 11:44

AbeJellinek requested changes Jan 22, 2024

View reviewed changes

JAC28 mentioned this pull request Oct 11, 2024

Extend Citavi6 translator to also import BibTex-Keys #3371

Closed

JAC28 reviewed Oct 11, 2024

View reviewed changes

JAC28 added a commit to JAC28/ZoteroTranslators that referenced this pull request Oct 11, 2024

Kombination of zotero#3183 and additional functions to import image f…

3963f03

…ilenames

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle custom fields in Citavi 5 XML.js #3183

Handle custom fields in Citavi 5 XML.js #3183

zuphilip commented Nov 9, 2023 •

edited

Loading

zuphilip commented Nov 28, 2023

zuphilip commented Jan 18, 2024 •

edited

Loading

adam3smith commented Jan 21, 2024 •

edited

Loading

AbeJellinek Jan 22, 2024

adam3smith Jan 22, 2024

zuphilip Jan 23, 2024 •

edited

Loading

zuphilip commented Jan 23, 2024

adam3smith commented Jan 27, 2024

zuphilip commented Jan 29, 2024

JAC28 left a comment

JAC28 Oct 11, 2024

JAC28 Oct 11, 2024

JAC28 Oct 11, 2024

JAC28 Oct 11, 2024

-	}
+	}
+	if (address) {
+		var addressMap = JSON.parse(address);
+		// Based on Demo project mapping is:
+		// 1->File
+		// 5->Link
+		if (addressMap.LinkedResourceType==1){
+			noteObject.note += "<p></p><p>" + '<strong>Missing Image with filename:</strong> </p>';
+			noteObject.note += "<pre><code>" +  addressMap.UriString + "</code></pre>\n";
+			var hasMissingImage = true;
+		}
+	}

-	}
+	}
+	if (hasMissingImage) {
+		noteObject.tags.push("#missingImage");
+	}

Handle custom fields in Citavi 5 XML.js #3183

Are you sure you want to change the base?

Handle custom fields in Citavi 5 XML.js #3183

Conversation

zuphilip commented Nov 9, 2023 • edited Loading

zuphilip commented Nov 28, 2023

zuphilip commented Jan 18, 2024 • edited Loading

adam3smith commented Jan 21, 2024 • edited Loading

AbeJellinek Jan 22, 2024

Choose a reason for hiding this comment

adam3smith Jan 22, 2024

Choose a reason for hiding this comment

zuphilip Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

zuphilip commented Jan 23, 2024

adam3smith commented Jan 27, 2024

zuphilip commented Jan 29, 2024

JAC28 left a comment

Choose a reason for hiding this comment

JAC28 Oct 11, 2024

Choose a reason for hiding this comment

JAC28 Oct 11, 2024

Choose a reason for hiding this comment

JAC28 Oct 11, 2024

Choose a reason for hiding this comment

JAC28 Oct 11, 2024

Choose a reason for hiding this comment

zuphilip commented Nov 9, 2023 •

edited

Loading

zuphilip commented Jan 18, 2024 •

edited

Loading

adam3smith commented Jan 21, 2024 •

edited

Loading

zuphilip Jan 23, 2024 •

edited

Loading