Integrating Matillion ETL with Amazon Simple AD
In this article we discuss the Amazon Simple AD service and configuring Matillion to use this service for User Authentication and Authorization. Whilst this blog post focuses on Amazon Simple AD, any LDAP compatible service may be used.
Amazon Simple AD is a standalone managed directory that is compatible with Microsoft Active Directory and is powered by Samba 4 Active Directory Compatible Server. Simple AD is just one of of various Directory services offered by AWS. Read about other directory-service offerings from Amazon here.
With Simple AD, you can centrally manage user accounts and group memberships for Amazon EC2 instances joined to a domain. It also allows you to use a single set of credentials to login across all EC2 instances as well as provide authentication to your applications. You may choose to make the service public and as with Matillion, authenticate from other devices and services.
Please note that whilst its very easy to create Simple AD directory service, Amazon does not provide any tools or interface to manage this service. Ensure that you have an EC2 Windows instance that is joined to the Simple AD domain once it’s created. You may then install AD Administration tools from Microsoft to manage users and usergroups.
Matillion runs on Apache Tomcat and leverages its support for JNDI Directory Realm to allow for LDAP integration. Read more about it here.
Authentication and Authorisation
Authentication is the process of verifying if a user is who they claim to be and Authorisation determines what features the user has access to. The user is authenticated by passing username/password to the LDAP service and then authorised by verifying if the user is part of certain usergroup’s in LDAP.
Matillion supports three roles to authorise and allow a user to access specific aspects of the product.
- Emerald: This role allows access to the ETL interface. Typically all users have this role
- Admin: This role allows a user to access the admin page
- API: this role allows a user to use Matillion ETL API
To support authorisation, we identify or create usergroup’s in LDAP and add users to these usergroups. A user may acquire one or more roles as necessary.
We’ll start by creating a Simple AD Domain and create users and usergroups as required. Then configure the appropriate options in Matillion.
Here’s the basic LDAP setup we will work with:
Step 1 and 2 below will create a Simple AD service with basic users and usergroups and Step 3 will configure LDAP in matillion admin.
Step 1 Create Simple AD Directory (test.mtln.com)
This step is optional and assumes you do not already have an LDAP service. Otherwise, proceed to Step 2.
Creating a Simple AD service is a well documented process. Please following the below links and setup a Simple AD Directory service – test.mtln.com.
A screenshot of the resulting service:
Please note, unless you create an explicit DNS entry, the Matillion server may not be able to reach domain test.mtln.com by name. The DNS Addresses for your directory service may be used instead.
At the time of writing this article, Simple AD did not support SSL communication.
Step 2 Create/Identify and Manage Users and Usergroups
As advised previously, you need a Windows PC to Manage Simple AD. I used a windows EC2 instance to create necessary users and usergroups. The following links take you to instructions about how to install and use Active Directory Users and Computers on EC2 instances running Microsoft Windows:
I created 3 usergroups (Emerald, Emerald Admin, Emerald API) in the Users Organizational Unit (OU) to represent the 3-roles supported by Matillion.
I then created 3 users, 1 per role. The following table shows the usergroups and their group-membership.
Here’s a screenshot of the final setup. Please note the users are in the Matillion Users OU whereas the usergroups are in Users OU. This was done to keep Matillion users separate from the rest. You may have them all under the Users OU.
Step 3 Configure LDAP in Matillion
There is already a detailed article on this topic. Please visit the following link for further information on configuring LDAP in Matillion.
Once tomcat is restarted, the user may now use their AD username and password to login. Just use your username. It is not required to specify the domain name.
LDAP integration in Matillion helps move user management and maintenance to an external directory service which is better suited for these purposes.
Simple AD from AWS can be setup within minutes and can be used to centrally manage users and security.
Hopefully this article gave you enough information to help you set up Simple AD and then use it for authenticating users in Matillion. If you have any questions or need further help, please contact us at – firstname.lastname@example.org
Download our free eBook for loads of useful advice and best practices on how to optimize your Amazon Redshift setup