{"id":119,"date":"2017-04-21T15:27:02","date_gmt":"2017-04-21T14:27:02","guid":{"rendered":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/?p=119"},"modified":"2017-04-28T10:09:41","modified_gmt":"2017-04-28T09:09:41","slug":"identifying-gaps-in-your-data","status":"publish","type":"post","link":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/2017\/04\/21\/identifying-gaps-in-your-data\/","title":{"rendered":"Identifying gaps in your data"},"content":{"rendered":"

One of the first things you want to do when you explore a new dataset is to identify possible gaps. Sample size and the number of variables are relevant but…how many observations do you have for each variable<\/em>? This distinction is even more relevant for archaeologists because (if we are being honest…) most of our data has huge gaps.<\/p>\n

Just to make the post clear:
\n– The Sample<\/em> is the set of entities you collected.
\n– Variables<\/em> are measures and properties of this sample
\n– Observations<\/em> are the values of the variables for each item in your sample<\/p>\n

The identification of variables with a decent number of observations is crucial for several processes. Let’s say that you have a bunch of archaeological sites and you want to create a map where the size of each dot (i.e. site<\/em>) is proportional to the area of the site. This would be a bad idea if 90% of your sample does not have an assigned area because these points will be ignored.<\/p>\n

This is even more relevant if you want to do some modelling (e.g. a linear regression). Lots of statistical models ignore variables that have observations with unassigned values so you have to be very careful about it. Let’s see how can we explore this issue.<\/p>\n

Example: Arrowheads in UK<\/h1>\n

Loading the dataset<\/h2>\n

We downloaded a dataset of arrowheads collected by the Portable Antiquities Scheme<\/a>:<\/p>\n[generic linenumbers=”False”]\narrowheads <- read.csv("https:\/\/git.io\/v9JJd", na.string="")
\n[\/generic]\n

As you can see I specified that empty strings of text should be read as NA<\/em> (i.e. Not Available). If you don’t do that then it will be read as an empty string, which is different than not having a value.<\/p>\n

If we take a look at the newly created arrowheads<\/em> data frame we will see a bunch of interesting metrics:<\/p>\n[generic linenumbers=”False”]\nstr(arrowheads)
\n[\/generic]\nYou should get something like:<\/p>\n[generic linenumbers=”False”]\n‘data.frame’: 1079 obs. of 13 variables:
\n $ id : int 522443 179174 233283 199204 106547 45059 508485 646649 401936 133638 …
\n $ classification : Factor w\/ 81 levels “Arrowhead”,”Barb and Tanged”,..: NA NA 10 NA NA NA NA NA NA NA …
\n $ subClassification: Factor w\/ 19 levels “barbed and tanged”,..: NA NA NA NA NA NA NA NA NA NA …
\n $ length : num 28 55.2 45.3 100.3 39 …
\n $ width : num 18 11 25 7.6 11 …
\n $ thickness : num 3 2 5.11 6.7 NA 3.54 3.47 6.5 1 NA …
\n $ diameter : num 2.2 4.5 4.58 5.1 6 6.82 6.96 7.5 8 8 …
\n $ weight : num NA 6.1 2.78 16.69 4.76 …
\n $ quantity : int 1 1 1 1 1 1 1 1 1 1 …
\n $ broadperiod : Factor w\/ 12 levels “BRONZE AGE”,”EARLY MEDIEVAL”,..: 1 4 1 4 1 4 4 4 4 4 …
\n $ fromdate : int -2150 NA -2150 1250 -2150 1066 1066 1200 1066 1200 …
\n $ todate : int -1500 NA -1500 1499 -800 1350 1400 1300 1500 1499 …
\n $ district : Factor w\/ 188 levels “Arun”,”Ashford”,..: 45 181 74 NA 26 93 177 137 186 53 …
\n[\/generic]\n

See all these NA<\/em> values? These are the gaps in our data. We can suspect that diameter<\/em> or “subClassification* will not be popular, but having over 1000 arrowheads it is difficult to know what variables should we be used in the analysis.<\/p>\n

Visualizing gaps<\/h2>\n

How can you identify these gaps? My preferred method is to visualize them using the Amelia package<\/a> (yes, awesome name<\/a> for an R package on missing data…). Its use is straightforward:<\/p>\n[generic linenumbers=”False”]\ninstall.package(“Amelia”)
\nlibrary(“Amelia”)
\nmissmap(arrowheads)
\n[\/generic]\n

\"\"
Missingness map<\/figcaption><\/figure>\n

The structure is R-classic: rows are sample units while columns are variables. Red cells are the ones that have some values while the other ones are empty.<\/p>\n

Interpretation<\/h2>\n

The map of missing values allows us to make informed decisions on how to proceed with the analysis. In this case:
\n– We should not use diameter<\/em> for analysis because it is not present in most of the sample.
\n– We have a almost complete information on broad spatial and temporal coordinates (broadperiod<\/em> and district<\/em>)
\n– classification<\/em> and subClassification<\/em> are quite useless here
\n– Measures that can be used are: weight<\/em>, thickness<\/em>, width<\/em> and length<\/em><\/p>\n

Impact<\/h2>\n

You can easily visualize the impact by creating a visualization with diameter<\/em> and another one without this value:
\n[generic linenumbers=”False”]\nggplot(arrowheads, aes(x=width, y=diameter, col=broadperiod)) + geom_point() + theme_bw() + facet_wrap(~broadperiod)
\n[\/generic]\n

\"\"<\/a>
Scatterplot width vs diameter<\/figcaption><\/figure>\n

Not looking good…R even tells you that you lost 1051 points in your dataset…also, most of the periods. Compare it with:<\/p>\n[generic linenumbers=”False”]\nggplot(arrowheads, aes(x=width, y=length, col=broadperiod)) + geom_point() + theme_bw() + facet_wrap(~broadperiod)
\n[\/generic]\n

\"\"<\/a>
Scatterplot width vs length<\/figcaption><\/figure>\n

Only 84 rows contained missing values, much better!<\/p>\n","protected":false},"excerpt":{"rendered":"

One of the first things you want to do when you explore a new dataset is to identify possible gaps. Sample size and the number of variables are relevant but…how many observations do you have for each variable? This distinction is even more relevant for archaeologists because (if we are being honest…) most of our […]<\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[16,15,18,19,17],"class_list":["post-119","post","type-post","status-publish","format-standard","hentry","category-r","tag-arrowheads","tag-dataviz","tag-eda","tag-hint","tag-pas"],"_links":{"self":[{"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/posts\/119","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/comments?post=119"}],"version-history":[{"count":5,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/posts\/119\/revisions"}],"predecessor-version":[{"id":166,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/posts\/119\/revisions\/166"}],"wp:attachment":[{"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/media?parent=119"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/categories?post=119"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/research.shca.ed.ac.uk\/past-by-numbers\/wp-json\/wp\/v2\/tags?post=119"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}