Skip to content

Commit

Permalink
Metadata2KG Round 2 instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
@oktie
oktie committed May 24, 2024
1 parent ba06709 commit 75bdb7f
Show file tree
Hide file tree
Showing 2 changed files with 134 additions and 119 deletions.
2 changes: 1 addition & 1 deletion data/metadata2kg/round2/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Metadata to KG Track Round 2 Datasets

In this round, a JSONL file is provided with each line representing a column in a table, along with table name, column name, and other columns in the same table. The goal is to map each such column to one "business glossary" item. We have also provided the metadata as well as the glossary in the form of an OWL ontology, to facilitate the mapping using ontology matching tools.
In this round, a JSONL file is provided with each line representing a column in a table, along with table name, column name, and other columns in the same table. The goal is to map each such column to one "glossary" item. We have also provided the metadata as well as the glossary in the form of an OWL ontology, to facilitate the mapping using ontology matching tools.

Sample data:
- [Sample Metadata File in JSONL](r2_sample_metadata.jsonl)
Expand Down
251 changes: 133 additions & 118 deletions docs/tracks/metadata-to-kg-track.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,137 +118,152 @@ <h2 class="text-2xl font-bold animation-delay-1000 mb-3">Round 1</h2>
JSONL file with each line containing a mapping of a column ID to an array of DBpedia property URIs and scores,
which will be sorted in descending order by score for evaluation. Round 1 data has one mapping for each column,
which is the most relevant property it maps to (e.g., if the column is about movie directors, the correct
mapping should be https://dbpedia.org/ontology/director). In Round 2, each column may map to more than one
property/class, or no property/class at all. The provided evaluation script measures Hit@1 and Hit@5. Other
measures may be added for final evaluation and in Round 2.</p>
mapping should be https://dbpedia.org/ontology/director). </p>
</main>

<main>
<section class="py-6 container">
<h2 class="text-2xl font-bold animation-delay-1000 mb-3">Round 2</h2>
<p class="leading-relaxed text-base">Round 2 dataset will consist of metadata of a number of relational tables and
a custom ontology. Stay tuned.
</p>
</main>

<main>
<section class="py-6 container">
<h2 class="text-2xl font-bold animation-delay-1000 mb-3">Submission Instructions</h2>
<p><strong>Submission</strong>: Are you ready? Then, submit up to 4 result sets for the <strong>test</strong>
set using the <a href="https://forms.gle/3RBxSSGR54zbdkfq7">Submission Form</a>.</p>
</main>


<!-- Co-Chairs -->
<section class="py-6 bg-grid bg-left-bottom container">
<h2 class="text-2xl font-bold bg-title animation-delay-75 mb-3">Track Organizers</h2>
<ul class="grid xl:grid-cols-4 lg:grid-cols-3 md:grid-cols-2 gap-2">

<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="mailto:[email protected]" class="p-4 flex flex-col">
<strong>Oktie Hassanzadeh</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
[email protected]
</span>
</a>
</li>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="mailto:[email protected]" class="p-4 flex flex-col">
<strong>Elita Lobo</strong>
<span class="text-slate-500">University of Massachusetts Amherst</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
[email protected]
</span>
</a>
</li>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="https://www.linkedin.com/in/nandanamihindu/" class="p-4 flex flex-col">
<strong>Nandana Mihindukulasooriya</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
nandana at ibm.com
</span>
</a>
</li>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="https://nhanph.github.io/" class="p-4 flex flex-col">
<strong>Nhan H. Pham</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
https://nhanph.github.io/
</span>
</a>
</li>
</ul>
<section class="py-6 container">
<h2 class="text-2xl font-bold animation-delay-1000 mb-3">Round 2</h2>
<p class="leading-relaxed text-base">
<a href="https://github.com/sem-tab-challenge/2024/tree/main/data/metadata2kg/round2">Round 2 dataset</a>
consists of a select set of open data table metadata
that need to be mapped to a custom glossary (dictionary of term labels and descriptions).
</p>
</section>
<p>Check out the <a
href="https://github.com/sem-tab-challenge/2024/tree/main/data/metadata2kg/round2/README.md">README file</a>
for input/output format, sample input/output, and an evaluation script. Note that the output of the mapping is a
JSONL file with each line containing a mapping of a column ID to an array of glossary items and scores,
which will be sorted in descending order by score for evaluation. Similar to Round 1 data, Round 2 data has one
mapping for each column,
which is the most relevant glossary item it maps to. We acknowledge that there may be more than one relevant
glossary item suitable for each column, which is why we use Hit@k scores for evaluation. We may use additional
scores that are not included in the evaluate.py script for the final ranking of the submissions.</p>
</main>


<aside id="date-card-container"
class="sticky bottom-0 sm:bottom-6 w-full sm:left-auto sm:right-6 sm:max-w-fit sm:ml-auto p-4 md:p-8 rounded-t-lg sm:rounded-lg bg-card border shadow-lg text-sm md:text-base transition-spacing duration-300">
<main>
<section class="py-6 container">
<h2 class="text-2xl font-bold animation-delay-1000 mb-3">Submission Instructions</h2>
<p><strong>Submission</strong>: Submit up to 4 result sets for the <strong>test</strong>
set using the following forms. You can participate in either round or both rounds:
<ul>
<li> <a href="https://forms.gle/3RBxSSGR54zbdkfq7"> Round 1 Submission Form</a> </li>
<li> <a href="https://forms.gle/fpp6dkYnR7WpE9rz7"> Round 2 Submission Form</a> </li>
</ul>
</p>
</main>


<label for="checkbox-date-card" class="cursor-pointer flex items-center justify-between">
<div class="flex gap-2 items-center">
<h2 class="text-2xl font-bold bg-title animation-delay-300">Important Dates</h2>
<span class="flex w-3 h-3 me-3 bg-red-500 rounded-full animate-pulse"></span>
</div>
<input checked="false" data-collapse-toggle="false" type="checkbox" id="checkbox-date-card"
name="checkbox-date-card" class="hidden peer" />
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-chevron-up h-6 w-6 peer-checked:rotate-180 rotate-0 transition-transform duration-300">
<path d="m18 15-6-6-6 6" />
</svg>
</label>
</div>
<!-- Co-Chairs -->
<section class="py-6 bg-grid bg-left-bottom container">
<h2 class="text-2xl font-bold bg-title animation-delay-75 mb-3">Track Organizers</h2>
<ul class="grid xl:grid-cols-4 lg:grid-cols-3 md:grid-cols-2 gap-2">

<div class="grid grid-rows-1fr mt-4 transition-grid-template-rows duration-300" id="date-card">
<ul class="list-disc pl-5 overflow-hidden">
<li>First Call for Challenge Participants: <span class="font-bold text-gray-400 line-through">March 30,
2024</span></li>
<li>Round 1 data release: <span class="font-bold text-gray-400 line-through">April 15, 2024</span>.</li>
<li>Round 2 data release: <span class="font-bold text-primary">May 27, 2024</span>.</li>
<li>Submission deadline for both rounds: <span class="font-bold text-primary">June 27, 2024</span></li>
<li>Paper Submission Deadline (Systems and Datasets): <span class="font-bold text-primary">July 15, 2024</span>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="mailto:[email protected]" class="p-4 flex flex-col">
<strong>Oktie Hassanzadeh</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
[email protected]
</span>
</a>
</li>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="mailto:[email protected]" class="p-4 flex flex-col">
<strong>Elita Lobo</strong>
<span class="text-slate-500">University of Massachusetts Amherst</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
[email protected]
</span>
</a>
</li>
<li>Participation invitations to present at ISWC 2024: <span class="font-bold text-primary">July 25, 2024</span>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="https://www.linkedin.com/in/nandanamihindu/" class="p-4 flex flex-col">
<strong>Nandana Mihindukulasooriya</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
nandana at ibm.com
</span>
</a>
</li>
<li>Submission of Final Papers: <span class="font-bold text-primary">September 15, 2024</span></li>
<li>Challenge presentation during OM workshop: <span class="font-bold text-primary">November 6-10, 2024</span>
<li class="border rounded-md bg-card hover:bg-card-hover">
<a href="https://nhanph.github.io/" class="p-4 flex flex-col">
<strong>Nhan H. Pham</strong>
<span class="text-slate-500">IBM Research</span>
<span class="text-sm flex items-center gap-1 mt-2">
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-mail">
<rect width="20" height="16" x="2" y="4" rx="2" />
<path d="m22 7-8.97 5.7a1.94 1.94 0 0 1-2.06 0L2 7" />
</svg>
https://nhanph.github.io/
</span>
</a>
</li>
<li>Challenge presentation during ISWC: <span class="font-bold text-primary">November 6-10, 2024</span></li>
</ul>
</div>
</p>
</section>


<aside id="date-card-container"
class="sticky bottom-0 sm:bottom-6 w-full sm:left-auto sm:right-6 sm:max-w-fit sm:ml-auto p-4 md:p-8 rounded-t-lg sm:rounded-lg bg-card border shadow-lg text-sm md:text-base transition-spacing duration-300">


<label for="checkbox-date-card" class="cursor-pointer flex items-center justify-between">
<div class="flex gap-2 items-center">
<h2 class="text-2xl font-bold bg-title animation-delay-300">Important Dates</h2>
<span class="flex w-3 h-3 me-3 bg-red-500 rounded-full animate-pulse"></span>
</div>
<input checked="false" data-collapse-toggle="false" type="checkbox" id="checkbox-date-card"
name="checkbox-date-card" class="hidden peer" />
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"
stroke-linecap="round" stroke-linejoin="round"
class="lucide lucide-chevron-up h-6 w-6 peer-checked:rotate-180 rotate-0 transition-transform duration-300">
<path d="m18 15-6-6-6 6" />
</svg>
</label>
</div>

<div class="grid grid-rows-1fr mt-4 transition-grid-template-rows duration-300" id="date-card">
<ul class="list-disc pl-5 overflow-hidden">
<li>First Call for Challenge Participants: <span class="font-bold text-gray-400 line-through">March 30,
2024</span></li>
<li>Round 1 data release: <span class="font-bold text-gray-400 line-through">April 15, 2024</span>.</li>
<li>Round 2 data release: <span class="font-bold text-primary">May 27, 2024</span>.</li>
<li>Submission deadline for both rounds: <span class="font-bold text-primary">June 27, 2024</span></li>
<li>Paper Submission Deadline (Systems and Datasets): <span class="font-bold text-primary">July 15,
2024</span>
</li>
<li>Participation invitations to present at ISWC 2024: <span class="font-bold text-primary">July 25,
2024</span>
</li>
<li>Submission of Final Papers: <span class="font-bold text-primary">September 15, 2024</span></li>
<li>Challenge presentation during OM workshop: <span class="font-bold text-primary">November 6-10, 2024</span>
</li>
<li>Challenge presentation during ISWC: <span class="font-bold text-primary">November 6-10, 2024</span></li>
</ul>
</div>

</aside>
</aside>

<script src="../js/main.min.js" defer></script>
<script src="../js/main.min.js" defer></script>
</body>

</html>

0 comments on commit 75bdb7f

Please sign in to comment.