As a community, when we are discussing recommendations and best
practices for how data should be published and governed, there is a
natural tendency for people to focus on the types of data they are
most familiar with working with.
This leads to suggestions that every dataset should have an API,
for example. Or that every dataset should be available in bulk.
While good general guidance, those approaches aren’t practical in
every case. That’s because we also need to take into account a
variety of other issues, including:
- the characteristics of the dataset
- the capabilities of the publishing organisation and the funding
their have available
- the purpose behind publishing the data
- and
the ethical, legal and social contexts in which it will be
used
I’m not going to cover all of that in this blog post.
But it occurred to me that it might be useful to describe a set
of dataset archetypes, that would function a bit like user personas.
They might help us better answer some of the basic
questions people have around data, discuss recommendations
around best practices, inform workshop exercises or just test our
assumptions.
To test this idea I’ve briefly described ten archetypes. For
each one I’ve tried to describe some it’s features, identified some
specific examples, and briefly outlined some of the challenges that
might apply in providing sustainable access to it.
Like any characterisation detail is lost. This is not an
exhaustive list. I haven’t attempted to list every possible
variation based on size, format, timeliness, category, etc. But
I’ve tried to capture a range that hopefully illustrate some
different characteristics. The archetypes reflect my own
experiences, you will have different thoughts and ideas. I’d love
to read them.
The Study
The Study is a dataset that was collected to support a research
project. The research group collected a variety of new data as part
of conducting their study. The dataset is small, focused on a
specific use case and there are no plans to maintain or update it
further as the research group does not have any ongoing funded to
collect or maintain the dataset. The data is provided as is for
others to reuse, e.g. to confirm the original analysis of the data
or to use it on other studies. To help others, and as part of
writing some academic papers that reference the dataset, the
research group has documented their methodology for collecting the
data. The dataset is likely published in an academic data portal or
alongside the academic papers that reference it.
Examples: water quality samples, field
sightings of animals, laboratory experiment results, bibliographic
data from a literature review, photos showing evidence of plant
diseases, consumer research survey results
The Sensor Feed
The Sensor Feed is a stream of sensor readings that are produced
by a collection of sensors that have been installed across a city.
New readings are added to the stream at regular intervals. The feed
is provided to allow a variety of applications to tap into the raw
sensor readings.. The data points are as directly reported by the
individual sensors and are not quality controlled. The individual
sensors may been updated, re-calibrated or replaced over time. The
readings are part of the operational infrastructure of the city so
can be expected to be available over at least the medium term. This
mean the dataset is effectively unbounded: new observations will
continue to be reported until the infrastructure is
decommissioned.
Examples: air quality readings, car park
occupancy, footfall measurements, rain gauges, traffic light
queuing counts, real-time bus locations
The Statistical Index
The Statistical Index is intended to provide insights into the
performance of specific social or economic policies by measuring
some aspect of a local community or economy. For example a sales or
well-being index. The index draws on a variety of primary datasets,
e.g. on commercial activities, which are then processed according
to a documented methodology to generate the index. The Index is
stewarded by an organisation and is expected to be available over
the long term. The dataset is relatively small and is reported
against specific geographic areas (e.g. from The Register) to
support comparisons. The Index is updated on a regular basis, e.g.
monthly or annually. Use of the data typically involves comparing
across time and location at different levels of aggregation.
Examples: street safety survey, consumer price
indices, happiness index, various national statistical indexes
The Register
The Register is a set of reference data that is useful for
adding context to other datasets. It consists of a list of specific
things, e.g. locations, cars, services with an unique identifier
and some basic descriptive metadata for each of the entries on the
list. The Register is relatively small, but may grow over time. It
is stewarded by an organisation tasked with making the data
available for others. The steward, or custodian, provides some
guarantees around the quality of the data. It is commonly
used as a means to link, validate and enrich other datasets and is
rarely used in isolation other than in reporting on changes to the
size and composition of the register.
Examples: licensed pubs, registered doctors,
lists of MOT stations, registered companies, a taxonomy of business
types, a statistical geography, addresses
The Database
The Database is a copy or extract of the data that underpins a
specific application or service. The database contains information
about a variety of different types of things, e.g. musicians, the
albums and songs. It is a relatively large dataset that can be used
to perform a variety of different types of query and to support a
variety of uses. As it is used in a live service it is regularly
updated, undergoes a variety of quality checks, and is growing over
time in both volume and scope. Some aspects of The Database may
reference one or more Registers or could be considered as Registers
in themselves.
Examples: geographic datasets that include a
variety of different types of features (e.g. OpenStreetMap,
MasterMap), databases of music (e.g. MusicBrainz) and books (e.g.
OpenLibrary), company product and customer databases, Wikidata
The Description
The Description is a collection of a few data points relating to
a single entity. Embedded into a single web page, it provides some
basic information about an event, or place, or company.
Individually it may be useful in context, e.g. to support a social
interaction or application share. The owner of the website provides
some data about the things that are discussed or featured on the
website, but does not have access to a full dataset. The individual
item descriptions are provided by website contributors using a CRM
to add content to the website. If available in aggregate, the
individual descriptions might make a useful Database or
Register.
Examples: descriptions of jobs, events, stores,
video content, articles
The Personal Records
The Personal Records are a history of the interactions of a
single person with a product or service. The data provides insight
into the individual person’s activities. The data is a slice
of a larger Dataset that contains data for a larger number of
people. As the information contains personal information it has to
be secure and the individual has various rights over the collection
and use of the data as granted by GDPR (or similar local
regulation). The dataset is relatively small, is focused on a
specific set of interactions, but is growing over time. Analysing
the data might provide useful insight to the individual that may
help them change their behaviour, increase their health, etc.
Examples: bank transactions, home energy usage, fitness or sleep
tracker, order history with an online service, location tracker,
health records
The Social Graph
The Social Graph is a dataset that describes the relationships
between a group of individuals. It is typically built-up by a small
number of contributions made by individuals that provide
information about their relationships and connections to others.
They may also provide information about those other people, e.g.
names, contact numbers, service ratings, etc. When published or
exported it is typically focused on a single individual, but might
be available in aggregate. It is different to Personal Records as
its specifically about multiple people, rather than a history of
information about an individual (although Personal Records may
reference or include data about others). The graph as a whole
is maintained by an organisation that is operating a social network
(or service that has social features).
Examples: social networks data, collaboration
graphs, reviews and trip histories from ride sharing services,
etc
The Observatory
The Observatory is a very large dataset produce by a coordinated
large-scale data collection exercise, for example by a range of
earth observation satellites. The data collection is intentionally
designed to support a variety of down-stream uses, which informs
the scale and type of data collected. The scale and type of data
can makes it difficult to use because of the need for specific
tools or expertise. But there are a wide range of ways in which the
raw data can be processed to create other types of data products,
to drive a variety of analyses, or used to power a variety of
services. It is refreshed and re-released as required by the
needs and financial constraints of the organisations collaborating
on collecting and using the dataset.
Examples: earth observation data, LIDAR point
clouds, data from astronomical surveys or Large Hadron Collider
experiments
The Forecast
The Forecast is used to predict the outcome of specific
real-world events, e.g. a weather or climate forecast. It draws on
a variety of primary datasets which are then processed and anlysed
to produce the output dataset. The process by which the predictions
are made are well-documented to provide insight into the quality of
the output. As the predictions are time-based the dataset has a
relatively short “shelf-life” which means that users need to
quickly access the most recent data for a specific location or area
of interest. Depending on the scale and granularity, Forecast
datasets can be very large, making them difficult to distribute in
a timely manner.
Example: weather forecasts
Let me know what you think of these. Do they provide any useful
perspective? How would you use or improve them?
since 2005.
I received a number of very
interesting job offers when I
I will
have to get up to speed with a lot of new things, and the legal and
travel costs overhead for Talis is significant, so I hope I can
turn this into a smart investment for them as quickly as possible.
I'll even rename my blog if necessary... ;-) For those wondering
about the future of my other projects, I'll write about them in a
separate post soon.
We are happy to announce
that SEMANTiCS 2019 will host the 14th DBpedia Community Meeting at
the last day of the conference on September 12, 2019.
It was surprisingly
easy to do, with a few hints from Barney. I’ve sketched out what I
did below. If you make one, remember that it can leak out private
information – names in particular – and can also be pretty sweary,
though mine’s not said anything outright offensive (yet).
So, I wrote 
