Home

FileMap

FileMap is a file-based map-reduce system for data-parallel computation.

Why Map-Reduce?

The map-reduce method of parallel computing was introduced by Google and further popularized by open source implementations like Hadoop, Disco, and others. Map-Reduce is remarkable in its simplicity and scalability. Traditional parallel environments have been based either on explicit message-passing APIs or on the appearance of a global shared-memory system. In contrast, Map-Reduce provides a rigid data-flow model in which the user need only write discrete kernel functions that fit within that dataflow. This restrictive model does not support many communication patterns in parallel codes. However, it is sufficient for a large number of data-intensive computing tasks. In return for accepting these restrictions, programmers can write simple, serial functions that a map-reduce run-time can parallelize very effectively.

What is unique about FileMap among other Map-Reduce frameworks?

FileMap is developed around several characteristic themes:

1. File-based, rather than tuple-based processing.
   • Rather than having to provide routines that parse input data in to (key,value) pairs, your code can operate directly on input data files. If you want the first stage to be a canonicalization of data to (key,value) strings, you can still do that, but it’s not required
   • Binary files (like ’’pcap’’ files) need not be expensively canonicalized into a string representation before processing
2. Language & tool-chain agnostic.
   • Reuse existing domain-specific and POSIX file processing tools.
   • Processing elements are executable programs rather than Java classes like Hadoop. This means you can use existing tools like ’’grep’’, ’’awk’’, and “tcpdump” in your processing.
3. Don’t re-invent the wheel
   • Thin layer on top of a POSIX (Linux, Unix, MacOS, etc.) environment.
   • Use OpenSSH for network communication and authentication
   • Use existing file systems & access control systems
4. Caching of intermediate results
5. Data replication
6. Streaming jobs where processing continues on new data as soon as it arrives
   • Partial-file support coming soon
7. No privileged user access or software installation required.
   • A single Python script and non-root ssh access to each node is all that is required.
8. Support multiple coalitions of participating systems

Examples

Please see the Examples page.

Installation

No real “installation” is required.

1. Download the “fm” command (a Python script) to the computer you will use to launch computations.
2. Create a “filemap.conf” file describing the nodes you are using.
3. Run “fm init” once to prepare necessary directory structures on each node.
4. Copy “fm” to each node in the computation and make sure it is in your PATH on each node.

Dependencies

• Python
• OpenSSH >= 4.0