Topic Reader
Avalanchio Topic allows you store real time data. The data a stored in a durable queue with a default expiry of 7 days. You can the read the topic data using the following script.
from avalanchio.remote_agent.connectors.topic_utils import TopicReader
# Name of the topic
topic = "FirewallRaw"
# Add a group id for watermarking the last read marker
group_id = "group123"
def main():
topic_reader = TopicReader(topic=topic, group_id=group_id)
for r in topic_reader:
print(r)
if __name__ == "__main__":
main()
Set starting offset by time
When Topic reader reads data, it persists the last read position so that if you want to restart reading from the same topic, you actually start reading from the last read position. However, you can update the starting position.
You can do that by specifying epoch timestamp or by starting offsets.