Data Profiling and Data Cleansing (WS 2014/15) - tele-TASK

Data profiling is the set of activities and processes to determine the metadata about a given dataset. Profiling data is an important and frequent activity of any IT professional and researcher.It encompasses a vast array of methods to examine data sets and produce metadata. Among the simpler results are statistics, such as the number of null values and distinct values in a column, its data type, or the most frequent patterns of its data values. Metadata that are more difficult to compute usually involve multiple columns, such as inclusion dependencies or functional dependencies between columns. More advanced techniques detect approximate properties or conditional properties of the data set at hand. The first part of the lecture examines efficient detection methods for these properties.Data profiling is relevant as a preparatory step to many use cases, such as query optimization, data mining, data integration, and data cleansing.Many of the insights gained during data profiling point to deficiencies of the data. Profiling reveals data errors, such as inconsistent formatting within a column, missing values, or outliers. Profiling results can also be used to measure and monitor the general quality of a dataset, for instance by determining the number of records that do not conform to previously established constraints. The second part of the lecture examines various methods and algorithms to improve the quality of data, with an emphasis on the many existing duplicate detection approaches.

Recent Episodes
  • Profiling Linked Data
    Jan 29, 2015 – 01:13:08
  • Generic Entity Resolution with Swoosh
    Jan 26, 2015 – 00:44:04
  • Sorted Neighborhood Methods & Generic Entity Resolution with Swoosh
    Jan 19, 2015 – 01:25:48
  • Sorted Neighborhood Methods
    Jan 15, 2015 – 01:25:58
  • Similarity Measures & Generic Entity Resolution with Swoosh
    Jan 8, 2015 – 01:26:54
  • Similarity Measures
    Jan 5, 2015 – 01:29:06
  • Duplicate Detection
    Dec 18, 2014 – 01:30:07
  • Data Quality and Data Cleansing
    Dec 15, 2014 – 01:21:07
  • Dependency Checking, Approximate FDs, FD_Mine and DFD
    Dec 11, 2014 – 01:29:33
  • IND Detection on very many Tables
    Dec 4, 2014 – 00:41:02
  • Discovery of Conditional Unique Column Combination
    Dec 4, 2014 – 00:24:04
  • TANE
    Dec 1, 2014 – 01:28:46
  • Der Apriori Algorithmus, Discovering cINDs & Detecting Functional Dependencies
    Nov 24, 2014 – 01:24:57
  • SPIDER, Foreign Key Extraction & Conditional Inclusion Dependencies
    Nov 13, 2014 – 01:27:04
  • Detecting Inclusion Dependencies
    Nov 10, 2014 – 01:20:03
  • Unique Column Combinations
    Oct 27, 2014 – 01:02:12
  • Visualization, Next Generation Profiling & Profiling Challenges
    Oct 23, 2014 – 01:24:32
  • An Introduction to Data Profiling
    Oct 20, 2014 – 01:31:44
  • Introduction
    Oct 13, 2014 – 01:29:33
Recent Reviews
Similar Podcasts
Disclaimer: The podcast and artwork on this page are property of the podcast owner, and not endorsed by UP.audio.