Skip to content

Commit

Permalink
paper
Browse files Browse the repository at this point in the history
  • Loading branch information
FlyingWorkshop committed Mar 19, 2024
1 parent 06863e4 commit 20fe6af
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 7 deletions.
6 changes: 1 addition & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,4 @@ Compression is handled by the `Compressor` abstract type.

## [MultivariateStats.jl](https://juliapackages.com/p/multivariatestats) Wrappers

As a convenience, we provide several wrappers for compression schemes from MultivariateStats.jl.

# Limitations

`CompressedBeliefMDP` is built off [`GenerativeBeliefMDP`](https://juliapomdp.github.io/POMDPs.jl/stable/POMDPTools/model/#POMDPTools.ModelTools.GenerativeBeliefMDP) which means the reward returned is the reward for a _single state sampled from the belief_ rather than the expected cumulative discounted reward.
As a convenience, we provide several wrappers for compression schemes from MultivariateStats.jl.
7 changes: 7 additions & 0 deletions paper.bib
Original file line number Diff line number Diff line change
Expand Up @@ -122,3 +122,10 @@ @inproceedings{EPCA
volume = {14},
year = {2001}
}

@misc{epca-MATLAB,
title={E-PCA},
author={Guillaume de Chambrier},
url={https://github.com/gpldecha/e-pca},
year={2016}
}
9 changes: 7 additions & 2 deletions paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,16 @@ bibliography: paper.bib

# Summary

Partially observable Markov decision processes (POMDPs) are a common framework in reinforcement learning and decision making under uncertainty, with applications in medicine [@drugs], sustainability [@carbon], economics [@markets], aerospace [@planes], and more. Unfortunately, solving real-world POMDPs with traditional methods is often computationally intractable due to the "curse of dimensionality" [@AFDM]. One way of overcome the so-called "curse of dimensionality" is belief compression. By compressing the belief distribution (i.e., the probability that the POMDP solver is in any given state at a given time), we can focus computation on the relevant belief states and solve large POMDPs. Belief compression is particularly useful when a POMDP has many states, but uncertainty is concentrated or sparse [@Roy].
Partially observable Markov decision processes (POMDPs) are a common framework in reinforcement learning and decision making under uncertainty, with applications in medicine [@drugs], sustainability [@carbon], economics [@markets], aerospace [@planes], and more. Unfortunately, solving real-world POMDPs with traditional methods is often computationally intractable due to the so-called "curse of dimensionality" [@AFDM]. Belief compression [@Roy] is a powerful technique for overcoming this curse that's particular effective when uncertainty is sparse or concentrated.

# Statement of need

CompressedBeliefMDPs.jl is a Julia package [@Julia] for solving large partially observable Markov decision processes (POMDPs). It's a part of the JuliaPOMDPs ecosystem [@POMDPs.jl] and generalizes the belief compression model presented in @Roy. In particular, it exports a `CompressedBeliefMDP` and `CompressedSolver` that generalize the methods in @Roy.
CompressedBeliefMDPs.jl is a Julia package [@Julia] for solving large POMDPs in the JuliaPOMDPs ecosystem [@POMDPs.jl]. CompressedBeliefMDPs.jl exports an interface that generalizes the belief compression algorithm presented in @Roy. In particular, while @Roy describe belief compression using Poisson exponential-family PCA, CompressedBeliefMDPs.jl supports arbitrary compression techniques and function approximators. This flexibility enables development and application of new, more powerful compression approaches as research in this area continues to evolve.


CompressedBeliefMDPs.jl exports an interface that generalizes the specific belief compression algorithm using Poisson PCA presented in @Roy. It supports integration of arbitrary compression techniques and function approximators within the overall POMDP solving process. This flexibility enables development and application of new, more powerful compression approaches as research in this area continues to evolve.

As far as we are aware, no prior Julia or Python package implements POMDP belief compression; though there is a similar package in MATLAB [@epca-MATLAB].

# Example

Expand Down

0 comments on commit 20fe6af

Please sign in to comment.