Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TCP connections could be more efficient. #5

Open
griffinmilsap opened this issue Dec 22, 2022 · 1 comment
Open

TCP connections could be more efficient. #5

griffinmilsap opened this issue Dec 22, 2022 · 1 comment
Labels
ezmsg-core Relates to the ezmsg.core package

Comments

@griffinmilsap
Copy link
Collaborator

Given the following scenario:

  • A publisher to topic on machine A
  • >1 subscriber to topic on machine B, all in same process

Currently, all subscribers will receive the TCP transmission and each overwrite the cache with the same (identical) received message that was serialized, sent, received, deserialized and copied multiple times needlessly. Fixing this would involve giving MessageCache a socket/server that it could synchronize with other MessageCaches running
on other machines/processes. This greatly complicates and fundamentally changes the backend message passing design, and may actually degrade performance for SHM and local caching significantly.

WORKAROUND/DESIGN TIP:
This performance issue can be worked around by creating a re-publisher that receives TCP messages once, then re-publishes them into local process/cache such that other subscribers in this process have direct cache access to the
message that was serialized only once from the source process.

@griffinmilsap griffinmilsap added the ezmsg-core Relates to the ezmsg.core package label Dec 22, 2022
@griffinmilsap
Copy link
Collaborator Author

This bug was mostly addressed (for multiple topics in the same unit) by v3.1.2 with message caching. This issue might be fully-addressed by adding Collection Stream republishers as discussed in #11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ezmsg-core Relates to the ezmsg.core package
Projects
None yet
Development

No branches or pull requests

1 participant