EDS: Eumetsat Data Store

Some experimental python scripts to download products from Eumetsat.

Release notes

text files used in python scripts (defined in eds_defs.py)
eds.key	replace <consumer_key> and <consumer_secret>
eds.prd	example file to store product to download (if needed)
python gui product selector
eds_select.py	select product, add in product file (see get_product() )
python scripts: functions
eds_defs.py	defines variables with default values
eds_funcs.py	several functions for downloading	6-8-2021: some bug fixes 5-10-2022: support no unzip
python scripts: examples using above functions
eds_get_in_period.py	Example 1	6-8-2021: Added hour range, minute range 25-1-2022: adapted to allow more flexible ranges 18-9-2022: def. product nr 1 in eds.prd
eds_get_live.py	Example 2	adapted 5-10-2022: choice not to unzip
eds_make_movie.py	Example 3	Adapted 16-1-2024, made suitable for ramdisk usage
eds_process.py	contains action to do with downloaded file (used in example 3)	18-9-2022: xrit2pic_cmd started explicitely from $HOME/bin instead of just bin

Example 1: download predefined time period and interval

Example 2: Regularly download new data

Example 3: Create a movie with limited time.

• Download newest 8 files of a product
• Create a movie using xrit2pic; this is specified in eds_process.py.

This file has some optional arguments:

  -h, --help        show this help message and exit
  -dry              just show what will be done
  -verbose VERBOSE  0=quiet, 1=errors, 2=+warnings, >2=+info
  -nprod NPROD      Select product from eds.prd (line nr.)
  -product PRODUCT  define product name

The Pi to use depends on the processing (i.e., xrit2pic command):

• For processing using single-channel a first generation Pi1 is OK.
• For images genereted using multiple channels a Pi4 with at least 2 GBytes RAM is needed. This example runs fine on a Pi4 with 8 GBytes RAM, but 4 GB should also work; probably 2 GB should be enough too.

  Repeatedly running can best be done by scheduling, e.g. using cron (Linux):

• In a command shell type: crontab -e
• At the bottom you will see:

  # m h  dom mon dow   command
  

  Add a line:

  */5 *  *   *   *     python3 eds/eds_make_movie.py > eds.log
  

  (this is if scripts are in directory $HOME/eds) This will run the script each day every 5 minutes, needed for RSS. You can change that to:

  */15 *  *   *   *     python3 eds/eds_make_movie.py >> eds.log
  

  for MSG (which is updated ecvery 15 minutes). Longer intervals are possible, since 8 new files can be handled at a time.

  Note: If the first line contains:

  #!/usr/bin/python3
  

  and if eds_make_movie.py is made executable then you can remove 'python3' in the command.

Notes installation

• eds_install.zip
• xrit2pic_arm.zip

Installation:

xrit2pic_cmd runs on Ubuntu 64-bits

• Open a command shell and put the zip files at your home directory (probably /home/pi)
• Type unzip eds_install.zip
  
  • This will create a directory eds containing the python files.
  • And 2 files eds.key and eds.prd at your home directory.
    • In eds.key, using an editor, replace the 2 lines with your Eumetsat consumer key and secret password.
• Type unzip xrit2pic_arm.zip
  This will create a directory bin containing xrit2pic_cmd
  and a directory lib containing the extra needed 3 libs
  
• Still being in your home directory type: cp lib/*.so /lib/.

• eds_get_live.py needs extra programs to handle the data. Running this will fill your disk because it only downloads files, as long as it runs.
• eds_make_movie.py is the only one using xrit2pic_cmd. This is a ready-to-use script.

Notes ramdisk; creating a movie on a Pi

sudo mkdir /ramdisk

ram /ramdisk  tmpfs  defaults,size=500m,x-gvfs-show  0  0

In eds_make_movie.py define for the pathes e.g.:

eds_defs.path_download='/ramdisk/zipped'
eds_defs.path_unzipped='/ramdisk/unzipped'

If the repeat cycle, defined in crontab, is faster than the download speed then it is possible that multiple instances of eds_make_movie.py are running. This is not a problem as such, but needs then much more RAMDisk. To prevent this use following script:

#!/bin/bash
pgrep -c python3
if [ "$?" == "0" ] ; then
 # date > last_skipped.txt
else
#  date > last_done.txt
  python3 $HOME/eds/eds_make_movie.py
fi

A few adaptions are made to eds_make_python.py:

• example ramdisk added
• create download path 'path_download' if it doesn't exist (no need to create 'path_unzipped' explicitely)
• Unzipped zip-files are 'emptied'. They are needed for some run/download cycles to prevent unnecessary repeated download, but will get a size 0 to prevent occupying space in ramdisk.

Notes if something fails

• eds/eds_funcs.py
• eds/eds_defs.py
• eds/eds_process.py
• eds/eds_make_movie.py, may be executable
• for xrit2pic_cmd:
  • bin/xrit2pic_cmd, must be executable
  • /lib/libjpeg812.so, must be executable
  • /lib/libwvt.so, must be executable
  • /lib/libhdf5.so, must be executable