When comparing two terms, word embeddings overcome the problems of lexical approaches based on string similarity: terms such as “city” and “location” could be considered as being very different in terms of string matching, but in a word embedding space these two terms may be closely related and considered as highly similar.

Word embeddings is a technique that allows mapping words or phrases to vectors of real numbers, capturing semantic regularities in vector spaces. This measure leverages contextual word embeddings to compute the similarity between tables using column headings and cell contents. To carry out the retrieval and integration processes, a (semantic) similarity measure between tabular data has been defined. It is interesting to note that considering these operators provides the basis for including others widely used in data integration, such as filtering and sorting. The present study is focused on union and join operators, since they are specially relevant for data integration, facilitating the consumption of data from disparate sources. Once this set of similar tables is obtained, the second step aims to determine which specific columns should be involved in the final integration process. In the first one, the objective is to retrieve tabular datasets Footnote 1 that have similar columns and are therefore more likely to be integrated. This paper presents an application of neural network language models to support data scientists in finding relationships between data in order to retrieve and integrate relevant tabular data. finding out their relationships) before retrieving other data that are potentially useful for integration. Using previously unknown tabular open data is therefore a time-consuming task for data scientists, who have to properly understand the data (e.g. Although tables are widely used, it is difficult to extract and manipulate automatically their content. Tabular data is a common way of storing, classifying and disseminating information. Moreover, most open data available on the Web are published in open data portals as tabular data (i.e. In such a scenario, it is essential for data scientists to have tools to support the integration process.ĭata integration is even more complex within an open data scenario, since the data to be consumed are not known a priori and more effort is required to retrieve and understand them. join or union) to offer a unified view of a set of heterogeneous data from different sources.

cleansing, combination or normalisation) that uses operations (e.g. This task involves several types of transformations (e.g.

Importantly, data integration comprises more than 80% of the effort for a data scientist due to its complex nature. The goal of data science is to move from raw data to knowledge through different steps, including data integration, wrangling, analysis, and visualisation. In the data integration task, contextual models also improved the results on union operation compared to context-free approaches. The results revealed that contextual models significantly outperform context-free models and a traditional weighting schema in ad hoc table retrieval. Column headings and cell values were used as contextual information and their impact on each task was evaluated. A comprehensive analysis of their performance was carried out on both retrieving and integrating tabular datasets, comparing them with context-free models. This work is the first attempt to use contextual word embeddings in the whole pipeline of table search and integration, including for the first time their use in the join operation. In this work, both processes were carried out using a similarity measure based on contextual word embeddings, which allows finding semantically similar tables and overcome the recall problem of lexical approaches based on string similarity. This paper presents a new approach to retrieve and further integrate tabular datasets (collections of rows and columns) using union and join operations.

Contextual pdf search