Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create script that can connect to ftp server and pull data from the meter #2

Open
teknofire opened this issue Jan 26, 2024 · 2 comments
Assignees

Comments

@teknofire
Copy link
Member

Now that we have access to the meter via ftp, we can start developing some scripts to download the files to a local system.

There are a few things we want to make sure that the script handles

  1. It should only download new files that haven't been previously downloaded.
  2. It needs to be able to keep track of the dates/times from the meter for files that are downloaded.
  3. It should capture what meter the files came from, we might need to discuss how we label this for example do we have names for specific meters or do we need to use something else for this.
  4. It should capture md5sums/checksums for any files download so as they are copied around later we can verify the integrity of the files.

We need to determine if the files show up on the ftp if they are still being written to or only once they are complete and ready to be downloaded. We wouldn't want to download a partial event file. If we find out that a file can still be written to we may have to not download the "latest" set of even files and only capture up to the n-1 set of files.

@nicolemah99
Copy link
Collaborator

Per @dayne we should also take user/password as parameters in case we have the chance to change it or add users.

@nicolemah99
Copy link
Collaborator

nicolemah99 commented Jan 31, 2024

I started a branch in data-duck-STREAM for the script, its currently pretty barebones until I can get access to the meter.
(https://github.com/acep-uaf/data-ducks-STREAM/tree/sel-meter-ftp-download)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

3 participants