Simon Willison’s Weblog

Managing data

Managing Data

These notes cover chapter one of “Data Management” by Richard T. Watson.

Organisations generate, store and process large amounts of data. Data management is the management of this organisational memory.

Individuals use data management techniques as well. Todo lists, calendars and address books are all examples of external memories (as opposed to our brain which is internal memory). These thre examples also share some common characteristics—they provide a standard format for storing specific kinds of information and they are organised in a way which accomodates rapid retrieval of information. They also requrie a trade-off between speed and size—a pocket calendar can not contain as much detailed information as a desk calendar but is far more conenient.

Organisations use all kinds of methods for storing data—filing cabinets, notice boards, computer systems and people are all frequently used. Storage devices are organised for rapid data entry and retrieval. Various types of information systems can be used:

Transaction processing system
TPS—collects and stores data from routine transactions
Management information system
MIS—converts data from a TPS in to information for managing an organisation
Decision support system
DSS—supports managerial decision making by provifing models for processing and analysing data
Executive information system
—provides information helpful to senior management for strategic decision making and organisation performance monitoring
On-line analytical processing
OLAP—presents multidimensional logical views of data
Data Mining
Uses statistical analysis to uncover hidden trends and relationships in data

Desirable atributes of data stored by an organisation include:

The data can be accessed by more than one person at a time. Low-volatility data often has many copies printed and distributed, which can cause problems if changes are made.
The data can easily be moved to the people who need to access it
Data is a valuable resource and must be kept safe. Backups should also be made of valuable data.
The information must be reliable and precise
The data must be current and up-to-date, especially if it is time sensitive
The data must be appropriate to the decisions being made by the organisation

An organisations memory exists in many different places, including inobvious areas such as the organisations roles and culture. Successfully designing and applying a data management solution is therefore a highly complex wicked problem.


People are an essential part of an organisation’s memory. The organisational culture is the collection of beliefs, values and attitudes that affect the behaviour of organisation members. People establish social networks of contacts, remembering who they can go to with specific problems or to access specific information. These networks are rarely documented. Knowledge of how to use organisational memory (who to ask about what, where to look things up) is called metamemory, but is often referred to as learning the ropes. Standard operating procedures are also an important part of organisational memory as they help organisations remember how to perform routine activities.


Documents are a commn medium for storing organisational data, and can take many different forms such as memos, manuals and reports. Hypertext allows the creation of non-linear documents which help readers find the information they are looking for more rapidly, but take longer to prepare.


Organisations sometimes build mathematical models to describe their business, which can be placed in the broad category of DSS and used to aid the decision making process.

Organisations often try to capture the knowledge of their employees in the form of rules and semantic nets stored in a knowledge base, which is another type of organisational memory.

External memories

There are companies that provide access to information as a service to other organisations. This information adds to the external memory of a company.

Problems with Data Management systems

The same data can be stored in several different systems around an organisation. A classic example of this is a customer’s address, which may end up stored in various customer related databases.
Lack of data control
Poor management of data, usually because different data is managed by different departments with little consistency across a large organisation.
Poor interface
Poor interfaces to data management systems can make the data virtually useless to the people who need to access it.
Decision makers need fast access to the data they need. Poorly designed data management systems can lead to delays.
Lack of reality
The data stored by an organisation must be relevant to the real world. Often systems will not store or provide access to the information needed by decision makers.
Lack of data integration
Data is frequently spread across many systems in an organisation. Integrating it is a long and complex process, best handled in an incremental manner as new systems are developed.

A brief history of Data Management

  • 1950s—file systems
  • 1960s—Hierarchical DBMS
  • 1970s—Network DBMS, followed by Relational DBMS
  • 1990s—Object-oriented DBMS

Data, information and knowledge

Raw, unsummarised and unanalyzed facts. Data is of very little use to decision makers as it contains far too much detail. Before it can be used it must be converted in to information.
Data that has been processed in to a meaningful form.
The capacity to use information—requires education and experience.

One person’s information can be another person’s data, depending on the level of detail required. We finish with some atrocious ASCII-art.

             | Knowledge | \ Interpretation
 |------|    |-----------|  \
 | Data |\        |          \||----------|
 |------| \       | Request    | Decision |
           \      |          _ |----------|
Conversion  \|   \|/         /|
           |-------------|  /
           | Information | / Interpretation

This is Managing data by Simon Willison, posted on 30th September 2002.

Next: Languages and grammars

Previous: Aquarionics backups

Previously hosted at