Recipe 6: Exploring the Emerging Python Client

Concepts

Sending commands to Egeria via direct HTTP requests can quickly become tedious and unwieldy - its much nicer (and hopefully more productive) to use a popular programming language like Python with libraries of pre-built functions to handle this for us. So we've embarked on creating a rich python client for Egeria. Now as you have probably started to surmise, Egeria offers a very, very broad and deep set of capabilities with hundreds of APIs. It doesn't seem likely that a python user will need them all. So we are designing the python client to cover the things that we believe will be commonly wanted. We can use the pyegeria client for both administrative and functional use cases and in this recipe we will do a bit of both.

The python client will naturally evolve to meet the needs of the community - and the community will hopefully encourage this by sharing use cases, examples, and even library extensions and bug fixes. This is an open process for community driven open source development.

The design of pyegeria broadly follows the structure of Egeria itself. If you recall how we used swagger in recipe 3, we showed an image like:

Each of the top level headings (such as Administration Services - Server Configuration) is a module that contains many restFul APIs. In pyegeria we generally create a separate class for each of these modules and class methods for each of the APIs. There are some cases, where it makes more sense to split up a large Egeria module into multiple python classes. For instance, Administration Services - Server Configuration has dozens of methods - but many of them aren't commonly used. So this module has been split into a set of core services and full services. The full services inherits the methods from the core services.

The following table lists the current status of the pyegeria classes and how they map onto Egeria. You will note that we have started by implementing more of the administrative and configuration methods first and are now incrementally building out functional capabilities.

Our thinking around the python client has evolved over the course of the work. Initially, we thought that the python client would primarily be used by administrators and perhaps some data scientists. Hence we started with configuration and operations. But over the last six months, we have seen more of a shift towards using Python for data engineering and more importantly have discovered how useful Python can be in user interfaces. This led us to move towards exposing more capabilities through the OMVS (Open Metadata View Services) rather than the OMAS (Open Metadata Access Services). In future recipes we will explore some of these nuances further.

The bottom line, is that the pyegeria client can be useful to administrators, data engineers, data scientists and more generally can serve as a foundation for many different kinds of user interfaces. The library can be used in simple executable python scripts, long-running server side services, user interfaces and jupyter notebooks.

So, enough discussion - let's try it out!

1. Installing pyegeria

Installing pyegeria is simple. We can use the normal pip install pyegeria mechanism to download pyegeria from pypi.org (the standard home for python packages) and we're off. However, if you are a bit rusty on configuring a Python environment, here is what we do:

• Choose a location to install pyegeria and run some experiments - creating an empty folder to work in is useful.

• Install Python if you don't already have it installed - there are many ways to do this. I often just install it from python.org.

• (Recommended) Configure a virtual environment - there are many flavors of this - for simplicity we can just use the standard venv by typing:

We can then activate this environment by issuing:

• Install pyegeria using your preferred tools. There are several popular approaches such as conda, pipenv, pip, and poetry. To do this with pip we issue:\

Hopefully all went smoothly - if not, please let us know! Now that pyegeria is installed, lets give it a try!

2. Explore some simple examples

To try out pyegeria we first need to identify an Egeria platform that we can use - and hopefully one that has already been configured. For the purposes of this recipe, we will assume that we are using the environment that we set up in our previous recipes. So lets start up Egeria just as we did in Recipe 1. Once the Egeria platform has started, open a new terminal window and start up python:

We should now have a python prompt allowing us to enter python commands interactively - so lets try a few simple things:

When I execute this sequence I get the following:

So what did we do? Let's walk through each step (and add one more):

• import the pyegeria package into the python environment

• instantiate the Platform client object, specifying the name of the Egeria server and platform to connect to as well as the user connect as. The object is assigned to the variable p_client. Instantiating the object is similar to starting a session; we can issue multiple method requests using the same session.

• Invoke the get_platform_origin() method provided by the Platform class. This is a simple method that returns the version of Egeria that is running on the platform. We use it like a hello-world application to check that the platform is running and we can connect to it.

• This tells pyegeria that we are done with the session and allows the resources to be recovered. Good practice.

Now, to find out about the classes and methods are available in pyegeria, we can use Python's built-in help command. To give it a try, lets start with an overview - in the Python window try:

• get general information about pyegeria along with the classes. Note that class names that begin with "_" are primarily for internal use within the package - it is unlikely that you will need to use them directly.

• This lists the methods for the Platform class. You will notice the interface to the help command is paged, like the unix more command...

• Demonstrates how useful this paged help interface is...there are a lot of methods.

• Provides information on just the get_platform_origin method of Platform.

It can also be convenient to put Python into help mode by typing help() and then just typing in the object you want help on. Leave by issuing quit.

3. Exploring Further

There is a lot that we can do with the pyegeria. Let's continue with a few more examples within the interactive python environment. Since we already have a Platform session, lets try a few more methods.

• This lists the servers that are currently active on the platform.

• This lists all the servers that have known configurations on the platform - which includes active servers.

• Whoa.......if you gave this a try you might be surprised how much text spews forth! This function is retrieving the full active configuration document for the server as a JSON document. It is probably pretty hard to read - so lets try this again in a way that we makes it easier to read:

• Still a lot of information - but it should be much more readable - and certainly gives you an appreciation of some of the hidden sophistication within Egeria. We'll explore configuration details in a future recipe. Many of our pyegeria methods produce JSON output - and some also require JSON inputs. We've tried to keep simple where we can.

Ok - your probably getting tired of the Platform class - so lets try another just for fun. Lets play with RegisteredInfo. One of the interesting things about Egeria is that it is self documenting. Its capabilities can be easily extended - or slimmed down for particular deployments. We also make extensive use of Egeria's reference data capabilities to support Egeria functions. What all this means is that there are methods we can use to have Egeria tell us about itself. These methods will be put in the RegisteredInfo class. So far there are only a few methods, but you can already see how useful it is. We'll give it a go:

• What you should see is a nicely printed list of the different kinds of services you can have listed with the list_registered_svcs() method. If you want to list just the access-services you would type:

• You should see a large amount of JSON come up on the screen...but for this function, I've done something fun and provided an additional option to format the output for you (since this is meant to be an informational method). Let's try again with:

• You should now see a somewhat more intelligible table - that seems to look best if the terminal window is about 40 characters wide. You can play with some of the other method parameters and see what you get...

This brings us to a final example to show. It is simple. Exit python (you can use ctrl-d) and in your terminal window type:

server_status.py

You should see a nice little status table of the OMAG servers running on your Egeria platform with a status column. This little widget was built using the open source package called rich Introduction — Rich 13.6.0 documentation and the pyegeria client. More examples can be found in the pyegeria examples folder. However:

Recap

In this recipe we have explored the new pyegeria python client for Egeria. We installed pyegeria and executed a number of examples to demonstrate how to use it. Although we showed the examples through Python's interactive command line, we can of course easily create scripts and applications that intersperse pyegeria calls with user logic and other Python packages. Development of pyegeria is very active, with bug fixes and extensions added frequently. If you want to get involved and help out, we'd be delighted to help you get started. There are many ways to help including - coding, testing, documenting and demoing. We believe that pyegeria will be an attractive starting point for many Egeria users and would be interested in your feedback as well. After all, this is a community project - and we welcome you to participate.