"Aa!  Isamashii houmu rogo desu ne?"
With ColdStore you can

Rationales

What we think is needed (if not inevitable) is a way to store heterogenous (unruly, unpredictable, complexly structured) data so it can be accessed quickly for processing by programs. The quickest way to access data for processing is to store it in the form it's used. By using mmap(), under Unix, you can keep the data on disk in precisely the form it's processed in, transparently, and persistently.

Very few programs run once, produce only what you see now, and then go away. A whole class of applications produce data which is intended to be around for a long time (relative to a given run of the program.)

Currently, conventionally, the data's explicitly written out to disk. Usually in a special storage hardened form. Of course, it has to be read back in again next time you want to process it.

Audience

This system is likely to be of use to you if you're looking for a way to store heterogenous data, access and process it quickly, and you're a programmer. While it's possible to present this kind of facility to an end user, it's unlikely to make much difference to them: after all, that's what users expect of everything anyway. This system is designed to help programmers deliver it with most convenience (to the programmers, of course :)

Blue Sky

Virtual Reality Improve the performance of VRML systems by storing VRML objects without having to parse them: you can render and operate on them directly out of memory without reparsing.

Store your Quake Levels the same way.

Workspaces (Anyone else remember APL?)

Your tcl/Python/Java/Scheme/PERL program can store over a gigabyte of data `natively', live, in variables, arrays, dictionaries.

In future versions of interpreters, programs/scripts would never terminate, but merely freeze, their current state captured in the coldstore. Thawing such an interpreter would bring it back to the precise state it was in, at the time of its freezing. EPCKPT supports this right now, but tacitly assumes that implementations will outlive objects (if .so files change, the results are undefined.) ColdStore assumes the opposite.

You don't need a database interface when everything's already got a name, or stored in something that does. The name we give to an object in a programming language is a unique key, which translates to the data contained within it ... this is just like a database index returning a record, but there's no need for a `select', it's already there.

You don't need files (well, except for backup.)
  • Editors that never die.
  • Editors that work directly on the parse-trees of the languages you're composing code for.
  • Parsers that take input from those editors, feed back to them, and feed forward via directly stored Abstract Syntax Trees, to compilers generating code on the fly.
MUDs
MOOs
MUSHes
M*
Good way to store those objects.

ColdStore could almost have been designed to implement a M* :)

Document Stores

Detail

ColdStore is a gigabyte-scale persistent object store which provides:

Possible Uses

We're planning to use it to make large network-capable interoperable scripting-language workspaces restartable (possibly migrateable).

You might use it as:

Future

The near future of ColdStore holds:

Status

Rapidly Approaching Beta (Tue Aug 10 12:02:39 EST 1999).

Requirements

Installation

  1. Get it from here.
  2. Unpack it: tar xzvf coldstore.tgz
  3. Make it: cd coldstore; make
  4. Try it out: cd chaos; ./chaos

Note: flex++ should have come with flex. It's sufficient to just ln flex flex++.

Contacts

More Info

We've got a Whitepaper. It goes into some detail as to design, but it's sadly out of date.

License

Short Form: It's all GPL.

All component licenses are GPL, except where noted (a couple of marked derived works.)

License terms are contained in each directory, in a file named LICENSE. All references to LICENSE in the source are accompanied by an MD5 checksum.

The complete work, considered as a portmanteau, is licensed under the terms contained in coldstore/LICENSE (MD5 f5220f8f599e5e926f37cf32efe3ab68), and Copyright Colin McCormack and Philippe Hébrais.

Acknowledgements

A lot of people have advanced the work, some unwittingly, some by merely standing and waiting (and waiting,) some actively:

Directions, Concepts and Influences.

Colin McCormack
Co-Founder. Obsessing, coding, obsessing some more.
Phillipe Hébrais
Co-Founder. Turned vague rumblings about Spatial Locality of Reference into a breakthrough, designed and coded QVMM unbelievably quickly and well. Contributed his experience from other experimental language developments to the design of this toolkit.
Jordan B. Baker
Enthusiasm and Chaos
Jeremy FitzHardinge
Crucial assistance with Elf and mmap (he wrote it, after all.)
Li2CO3
Visual Artist in residence.
Andrew Morton
Gentle enquiries.
Ryan Daum
Suggesting we look carefully at Python. Needling us until we released.
Nick Sweeney
Countless conversations, numberless insights.
Bill Drury
The space in which we converse (xao.com 7777)
Greg Hudson
His epochal coldmud.
Hans Reiser
His epochal reiserfs.
Guido von Rossum
His orthogonal Python API.
Brian Eno
Oblique Strategies and Soundtrack